Oct 30 01:26:03.703347 ip-10-0-143-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 01:26:03.703359 ip-10-0-143-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 01:26:03.703366 ip-10-0-143-27 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 01:26:03.703642 ip-10-0-143-27 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 01:26:15.026571 ip-10-0-143-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 01:26:15.026591 ip-10-0-143-27 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 55071aca0de44905a831755f1c492b6c -- Oct 30 01:28:18.970420 ip-10-0-143-27 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:28:19.396019 ip-10-0-143-27 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:19.396019 ip-10-0-143-27 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:28:19.396019 ip-10-0-143-27 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:19.396019 ip-10-0-143-27 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401451 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401538 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401543 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401546 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401549 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401552 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.401544 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401555 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401558 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401561 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401563 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401566 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401569 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401572 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401575 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401577 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401581 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401585 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401587 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401590 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401593 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401595 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401598 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401601 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401604 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401606 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.401798 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401609 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401612 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401614 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401617 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401620 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401632 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401635 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401637 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401640 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401643 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401646 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401648 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401651 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401654 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401656 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401660 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401662 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401665 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401668 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401671 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.402255 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401674 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401676 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401679 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401682 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401686 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401692 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401695 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401698 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401701 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401704 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401706 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401709 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401711 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401714 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401716 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401719 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401722 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401724 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401727 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.402723 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401730 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401733 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401737 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401741 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401744 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401746 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401749 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401752 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401755 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401758 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401761 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401763 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401766 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401769 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401772 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401775 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401777 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401780 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401782 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.403189 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401785 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401790 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401793 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401796 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401798 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401802 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401805 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401807 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401810 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401813 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401815 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401818 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401821 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401824 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401827 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401830 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401832 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401834 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401837 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.403656 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401839 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.401844 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402654 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402662 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402665 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402667 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402670 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402673 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402676 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402679 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402681 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402684 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402687 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402690 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402692 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402695 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402699 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402702 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402704 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.404110 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402707 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402710 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402713 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402715 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402718 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402721 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402724 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402726 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402729 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402732 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402735 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402738 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402740 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402743 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402746 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402748 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402751 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402753 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402756 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402758 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.404604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402761 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402763 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402766 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402769 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402771 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402774 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402777 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402780 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402782 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402785 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402788 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402792 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402795 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402798 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402800 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402803 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402805 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402811 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402813 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.405073 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402816 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402820 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402823 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402825 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402828 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402830 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402833 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402836 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402838 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402841 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402843 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402846 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402849 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402851 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402855 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402857 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402859 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402862 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402865 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402867 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.405532 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402871 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402873 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402877 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402881 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402884 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402887 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402890 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402892 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402895 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402901 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402904 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402907 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402909 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402912 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402914 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402917 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402919 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402922 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.406003 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402925 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402927 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402930 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402933 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402935 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402938 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402940 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402943 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402945 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402949 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.402952 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403531 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403538 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403543 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403546 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403549 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403552 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403555 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403558 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403560 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.406487 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403563 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403566 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403568 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403573 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403576 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403578 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403581 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403583 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403586 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403589 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403592 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403595 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403597 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403600 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403605 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403607 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403610 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403613 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403615 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.406983 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403618 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403620 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403623 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403626 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403628 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403630 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403635 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403639 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403643 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403646 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403649 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403651 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403654 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403657 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403660 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403664 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403667 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403669 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403672 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.407477 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403675 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403677 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403680 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403682 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403685 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403687 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403690 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403692 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403695 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403698 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403700 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403703 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403705 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403708 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403710 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403713 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403716 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403718 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403722 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.407955 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403725 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403728 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403730 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403733 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403735 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403738 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403741 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403743 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403746 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403749 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403752 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403755 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403758 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403760 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403763 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403766 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403768 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403771 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403773 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403776 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.408415 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403778 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403781 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403783 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403786 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403789 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403791 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403796 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403798 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403801 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403803 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403809 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403812 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403814 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403819 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403823 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403827 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.403834 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403928 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:28:19.408877 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403935 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403942 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403947 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403953 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403956 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403961 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403965 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403968 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403971 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403975 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403978 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403981 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403984 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403987 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403990 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403993 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.403996 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404000 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404003 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404006 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404009 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404013 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404017 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:28:19.409394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404021 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404024 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404028 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404031 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404035 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404038 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404041 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404044 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404048 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404052 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404055 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404058 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404061 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404063 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404068 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404071 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404074 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404077 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404080 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404084 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404087 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404091 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404094 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404097 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404100 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:28:19.409932 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404103 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404106 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404108 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404111 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404114 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404131 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404134 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404138 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404142 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404145 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404148 2577 flags.go:64] FLAG: --help="false" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404151 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404155 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404158 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404161 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404164 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404167 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404171 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404173 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404176 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404179 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404182 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404185 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404188 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:28:19.410613 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404191 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404194 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404197 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404200 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404202 2577 flags.go:64] FLAG: --lock-file="" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404205 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404208 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404211 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404217 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404220 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404223 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404226 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404229 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404232 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404235 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404239 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404243 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404247 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404252 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404255 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404258 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404262 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404265 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404268 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404271 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:28:19.411286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404274 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404281 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404284 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404287 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404290 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404293 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404296 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404300 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404303 2577 flags.go:64] FLAG: --port="10250" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404306 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404308 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e80abfd091b34d36" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404311 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404323 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404326 2577 flags.go:64] FLAG: --register-node="true" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404329 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404333 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404336 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404339 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404342 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404346 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404349 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404352 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404356 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404359 2577 flags.go:64] FLAG: --runonce="false" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404363 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404367 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404370 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:28:19.411869 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404373 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404376 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404380 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404383 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404386 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404389 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404392 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404395 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404398 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404401 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404404 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404407 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404413 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404416 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404419 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404423 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404426 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404429 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404431 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404435 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404437 2577 flags.go:64] FLAG: --v="2" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404442 2577 flags.go:64] FLAG: --version="false" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404446 2577 flags.go:64] FLAG: --vmodule="" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404450 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404453 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:28:19.412610 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404584 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404587 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404591 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404594 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404598 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404601 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404603 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404606 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404608 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404611 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404615 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404618 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404622 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404626 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404629 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404632 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404635 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404637 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404640 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.413243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404642 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404645 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404648 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404650 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404653 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404656 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404658 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404661 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404663 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404665 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404668 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404670 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404673 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404676 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404678 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404682 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404684 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404689 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404693 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.413686 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404696 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404699 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404702 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404705 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404709 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404712 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404715 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404717 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404720 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404722 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404725 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404727 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404730 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404733 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404735 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404738 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404740 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404743 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404745 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404748 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.414163 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404750 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404753 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404756 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404758 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404761 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404764 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404766 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404769 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404773 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404775 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404779 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404785 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404787 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404790 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404795 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404798 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404801 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404803 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.414634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404806 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404809 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404811 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404814 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404816 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404819 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404821 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404824 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404827 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404829 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404832 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404835 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404837 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404840 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404845 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404847 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404850 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404852 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404855 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.415094 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404859 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404861 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404866 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404871 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404874 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404877 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404879 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.404906 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404940 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404945 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404949 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404952 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404955 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404958 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404960 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.415586 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404963 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404966 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404968 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404971 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404973 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404976 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404979 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404981 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404984 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404986 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404992 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404995 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.404997 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405001 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405003 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405008 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405010 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405013 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.415956 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405017 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405019 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405022 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405025 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405027 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405030 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405032 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405035 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405038 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405041 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405044 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405047 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405050 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405053 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405055 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405058 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405060 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405063 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405065 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405068 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.416447 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405070 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405073 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405076 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405078 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405081 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405083 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405086 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405089 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405092 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405095 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405098 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405100 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405104 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405107 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405109 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405112 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405114 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405117 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405131 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.416894 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405134 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405136 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405139 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405142 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405146 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405149 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405151 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405155 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405159 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405162 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405165 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405167 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405170 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405173 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405176 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405179 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405181 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405184 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405187 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.417353 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405189 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405192 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405195 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405198 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405200 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405203 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405207 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405209 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405212 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405215 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405217 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405220 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405223 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405226 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405230 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405234 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405237 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405241 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405244 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.417795 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405247 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405249 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.405252 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.412169 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.412184 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.412199 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.412203 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412262 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412267 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412271 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412274 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412277 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412280 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412283 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412285 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412288 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412291 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412294 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412296 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412299 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.418287 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412302 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412305 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412307 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412310 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412312 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412315 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412317 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412320 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412323 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412325 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412329 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412331 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412334 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412337 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412339 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412343 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412346 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412348 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412351 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.418753 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412354 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412356 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412359 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412362 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412364 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412367 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412370 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412372 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412375 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412377 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412380 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412383 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412386 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412388 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412391 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412394 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412396 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412399 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412401 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412404 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.419199 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412407 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412410 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412412 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412417 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412421 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412424 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412427 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412430 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412432 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412435 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412439 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412441 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412444 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412447 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412450 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412452 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412455 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412458 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412460 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.419674 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412464 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412468 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412471 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412473 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412476 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412479 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412481 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412484 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412486 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412489 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412491 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412494 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412496 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412499 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412502 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412505 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412507 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412510 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412514 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412517 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.420105 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412519 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412522 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412526 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412529 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412531 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412534 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412537 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412540 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412542 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412545 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412548 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412551 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412553 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412556 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412558 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.420634 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.412588 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412728 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412733 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412736 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412740 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412743 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412746 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412748 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412751 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412754 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412757 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412759 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412762 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412765 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412768 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412770 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412774 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412776 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412779 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.420992 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412784 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412787 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412789 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412792 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412795 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412797 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412800 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412802 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412805 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412808 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412810 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412813 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412816 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412818 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412821 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412823 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412826 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412829 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412831 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.421503 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412834 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412836 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412839 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412841 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412844 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412847 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412850 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412852 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412855 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412857 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412860 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412863 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412865 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412868 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412870 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412872 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412875 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412878 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412880 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412883 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.421938 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412885 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412889 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412893 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412896 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412900 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412902 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412905 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412908 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412910 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412913 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412915 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412918 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412920 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412923 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412926 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412928 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412931 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412934 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412936 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.422678 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412939 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412941 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412944 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412946 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412949 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412952 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412954 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412957 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412959 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412962 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412964 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412967 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412969 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412973 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412976 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412979 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412982 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412985 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412987 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.423349 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412989 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412992 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412994 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.412997 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413000 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413002 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413004 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413007 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413010 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413013 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.413035 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413065 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413070 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413073 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413076 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.423803 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413078 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413081 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413083 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413087 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413089 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413092 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413094 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413097 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413100 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413103 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413106 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413108 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413111 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413113 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413116 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413131 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413134 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413136 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413139 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413142 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.424243 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413144 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413147 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413150 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413152 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413155 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413157 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413160 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413162 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413165 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413168 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413170 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413173 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413175 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413178 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413181 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413183 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413186 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413189 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413193 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413195 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.424848 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413198 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413201 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413203 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413207 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413211 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413214 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413217 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413219 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413222 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413224 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413227 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413229 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413232 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413234 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413237 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413240 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413242 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413244 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413247 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.425762 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413250 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413253 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413256 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413259 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413261 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413264 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413269 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413271 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413274 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413276 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413279 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413282 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413285 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413287 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413290 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413294 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413297 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413300 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413303 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.426362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413306 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413308 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413311 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413313 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413315 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413318 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413320 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413323 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413325 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413328 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413331 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413333 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413336 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413338 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413341 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413343 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413346 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413349 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413351 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413355 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.426812 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413358 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413362 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413365 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:19.413367 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.413417 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.417751 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.418738 2577 server.go:1014] "Starting client certificate rotation" Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.418837 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.418873 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.420976 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:19.427301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.425929 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:19.438210 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.438191 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:28:19.441705 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.441687 2577 log.go:25] "Validated CRI v1 image API" Oct 30 01:28:19.442527 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.442500 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:28:19.446686 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.446659 2577 fs.go:135] Filesystem UUIDs: map[0f2a8570-a2fc-4e18-a79a-5f4b5c41f552:/dev/nvme0n1p4 2151f9e5-99a8-4728-8619-ed78212dacfb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 01:28:19.446734 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.446686 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 01:28:19.452133 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.452017 2577 manager.go:217] Machine: {Timestamp:2026-10-30 01:28:19.450395251 +0000 UTC m=+0.349191180 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100467 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a5001aabf7e286208b4dcc48af2a3 SystemUUID:ec2a5001-aabf-7e28-6208-b4dcc48af2a3 BootID:55071aca-0de4-4905-a831-755f1c492b6c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:5e:bc:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:5e:bc:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:6c:93:e6:1c:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:28:19.452211 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.452135 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:28:19.452261 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.452219 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:28:19.455871 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.455843 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:28:19.456015 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.455874 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:28:19.456056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.456017 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:28:19.456056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.456025 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:28:19.456114 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.456064 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:19.456114 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.456079 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:19.456912 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.456902 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:28:19.461200 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.461187 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:28:19.461312 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.461303 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:28:19.462264 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.462255 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:28:19.462302 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.462268 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:28:19.462302 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.462279 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:28:19.462302 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.462287 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:28:19.462302 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.462296 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:28:19.462637 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.462624 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:19.467470 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.467451 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:28:19.472674 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.472656 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:19.473500 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473486 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:28:19.473563 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473517 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:28:19.473689 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473678 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:28:19.473716 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473695 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:28:19.473716 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473702 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:28:19.473716 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473707 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:28:19.473716 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473713 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:28:19.473716 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.473718 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:28:19.474398 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.474389 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:28:19.474429 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.474399 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:28:19.474429 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.474407 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:28:19.474429 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.474413 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:28:19.474429 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.474423 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:28:19.474579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.474431 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:28:19.475690 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.475679 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:28:19.475734 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.475693 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:28:19.479667 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.479653 2577 server.go:1263] "Started kubelet" Oct 30 01:28:19.479790 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.479742 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:28:19.479835 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.479813 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:28:19.479877 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.479754 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:28:19.480524 ip-10-0-143-27 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:28:19.480976 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.480960 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:28:19.484724 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.484709 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:28:19.486228 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.486208 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 01:28:19.487402 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.487382 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:28:19.487498 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.487389 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:19.487937 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.487624 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-27.ec2.internal\" not found" Oct 30 01:28:19.488185 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488152 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:28:19.488185 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488187 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:28:19.488324 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488167 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:28:19.488324 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488299 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:28:19.488324 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488307 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:28:19.488640 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488624 2577 factory.go:55] Registering systemd factory Oct 30 01:28:19.488722 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488643 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:28:19.488872 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488857 2577 factory.go:153] Registering CRI-O factory Oct 30 01:28:19.488924 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488876 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 01:28:19.488977 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488927 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:28:19.488977 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488945 2577 factory.go:103] Registering Raw factory Oct 30 01:28:19.488977 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.488959 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 01:28:19.489827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.489807 2577 manager.go:319] Starting recovery of all containers Oct 30 01:28:19.498469 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.498429 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:28:19.500261 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.500237 2577 manager.go:324] Recovery completed Oct 30 01:28:19.502163 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.502109 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:28:19.504996 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.504974 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r6l24" Oct 30 01:28:19.507632 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.507029 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.509010 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.508986 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.509092 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.509027 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.509092 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.509039 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.509643 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.509626 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:28:19.509643 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.509640 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:28:19.509757 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.509661 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.509757 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.509661 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:28:19.510349 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.510333 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.510397 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.510362 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.510397 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.510372 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.512211 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.512199 2577 policy_none.go:50] "Start" Oct 30 01:28:19.512267 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.512213 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:28:19.512267 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.512224 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:28:19.514323 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.514311 2577 policy_none.go:44] "Start" Oct 30 01:28:19.515261 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.515236 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 01:28:19.515261 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.515252 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 01:28:19.516447 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.515249 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-27.ec2.internal.18e32a673ac59be5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-27.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-27.ec2.internal,},FirstTimestamp:2026-10-30 01:28:19.479616485 +0000 UTC m=+0.378412414,LastTimestamp:2026-10-30 01:28:19.479616485 +0000 UTC m=+0.378412414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-27.ec2.internal,}" Oct 30 01:28:19.520210 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.519982 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r6l24" Oct 30 01:28:19.534359 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.534279 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-27.ec2.internal.18e32a673c86222c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-27.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-27.ec2.internal,},FirstTimestamp:2026-10-30 01:28:19.509010988 +0000 UTC m=+0.407806919,LastTimestamp:2026-10-30 01:28:19.509010988 +0000 UTC m=+0.407806919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-27.ec2.internal,}" Oct 30 01:28:19.555883 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.555848 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:28:19.558524 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.558511 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 01:28:19.558601 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.558543 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:28:19.558601 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.558554 2577 server.go:87] "Starting device plugin registration server" Oct 30 01:28:19.558688 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.558669 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:19.558843 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.558831 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:28:19.558884 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.558845 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:28:19.558956 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.558939 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:28:19.559018 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.559008 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:28:19.559018 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.559018 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:28:19.559644 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.559623 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:28:19.559731 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.559668 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-27.ec2.internal\" not found" Oct 30 01:28:19.577167 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.577145 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:19.578459 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.578170 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:19.629426 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.629401 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:28:19.629426 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.629426 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:28:19.631396 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.629447 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:28:19.631396 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.629481 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 01:28:19.642425 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.642398 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:19.659835 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.659773 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.660735 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.660714 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.660838 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.660749 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.660838 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.660763 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.660838 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.660787 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.679818 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.679795 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.679818 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:19.679818 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-27.ec2.internal\": node \"ip-10-0-143-27.ec2.internal\" not found" Oct 30 01:28:19.730203 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.730164 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal"] Oct 30 01:28:19.732486 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.732472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.734229 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.734211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.770963 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.770942 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.773228 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.773213 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.789905 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.789882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.789991 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.789913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.789991 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.789929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fc72893a455f4d88649494269f87258-config\") pod \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" (UID: \"0fc72893a455f4d88649494269f87258\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.791551 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.791534 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:19.794752 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.794737 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:19.891056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.891006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.891056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.891053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.891270 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.891089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fc72893a455f4d88649494269f87258-config\") pod \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" (UID: \"0fc72893a455f4d88649494269f87258\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.891270 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.891106 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.891270 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.891113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:19.891270 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:19.891138 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fc72893a455f4d88649494269f87258-config\") pod \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" (UID: \"0fc72893a455f4d88649494269f87258\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:28:20.095566 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.095477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:28:20.098607 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.098580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:28:20.419165 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.418898 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 01:28:20.419753 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.419282 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:28:20.419753 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.419290 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:28:20.419753 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.419289 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:28:20.462817 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.462783 2577 apiserver.go:52] "Watching apiserver" Oct 30 01:28:20.487750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.487720 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:20.515412 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.515383 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:28:20.515734 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.515715 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9qlzc","kube-system/global-pull-secret-syncer-jt6kw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt","openshift-cluster-node-tuning-operator/tuned-s98cl","openshift-image-registry/node-ca-qmg28","openshift-multus/multus-gcqqc","kube-system/konnectivity-agent-lwf9z","openshift-network-operator/iptables-alerter-7c4wq","openshift-ovn-kubernetes/ovnkube-node-2kd25","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal","openshift-multus/multus-additional-cni-plugins-f9pb2","kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal","openshift-dns/node-resolver-5w55t","openshift-multus/network-metrics-daemon-xv85r"] Oct 30 01:28:20.518931 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.518912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:20.519042 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.519018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:20.520998 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.520973 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.521080 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.521060 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:20.523242 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.523224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.525438 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.525395 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:23:20 +0000 UTC" deadline="2028-05-11 01:21:19.244296612 +0000 UTC" Oct 30 01:28:20.525438 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.525435 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13415h52m58.718863592s" Oct 30 01:28:20.525982 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.525967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.528308 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.528286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.530480 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.530461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.532785 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.532765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.535761 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.535745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.538778 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.538755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.541367 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.541349 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:28:20.541516 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.541498 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.541929 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.541916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.543952 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.543933 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.544070 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.543983 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c6clm\"" Oct 30 01:28:20.544070 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.543986 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.544070 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.544025 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rwcx9\"" Oct 30 01:28:20.544070 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.544039 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.547824 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.544899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.547824 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.545869 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.547824 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.546063 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.547824 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.546258 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7p9xn\"" Oct 30 01:28:20.547824 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.546345 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:28:20.549184 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.549164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:20.549291 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.549235 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:20.551614 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.551597 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkdgr\"" Oct 30 01:28:20.590101 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.590073 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:28:20.596341 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-netns\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.596478 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-multus\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.596478 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cnibin\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.596597 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-tmp-dir\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.596654 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:20.596654 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-kubernetes\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.596739 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-run\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.596739 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-lib-modules\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.596739 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-bin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.596859 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-var-lib-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.596859 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-netns\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.596934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7pqj\" (UniqueName: \"kubernetes.io/projected/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-kube-api-access-h7pqj\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.596934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9x6\" (UniqueName: \"kubernetes.io/projected/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-kube-api-access-jg9x6\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.597009 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-host\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.597009 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.596976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-ovn\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.597112 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.597112 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.597220 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-sys-fs\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.597263 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-tmp\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.597299 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9712dd02-a8c8-402f-bfcd-8e149dc4835b-host\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.597340 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cnibin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.597376 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5kq\" (UniqueName: \"kubernetes.io/projected/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-kube-api-access-7p5kq\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.597409 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c859dda-f9c4-4c70-b6d2-e72b990dc465-agent-certs\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.597409 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-hosts-file\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.597477 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.597477 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-sys\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.597548 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.597548 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.597548 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.597654 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-kubelet\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.597654 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c859dda-f9c4-4c70-b6d2-e72b990dc465-konnectivity-ca\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.597654 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-netd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.597654 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-system-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.597788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-systemd-units\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.597788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.597788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-socket-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.597788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2c22\" (UniqueName: \"kubernetes.io/projected/7b706a15-4789-4900-a62b-ba4275904124-kube-api-access-j2c22\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.597958 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-etc-kubernetes\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.597958 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-bin\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.597958 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.598058 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-system-cni-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.598058 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.597998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.598058 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-device-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.598188 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-k8s-cni-cncf-io\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.598188 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-hostroot\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.598188 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:20.598359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-registration-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.598359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.598359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-multus-certs\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.598359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598293 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-slash\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.598359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-script-lib\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.598359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrnd\" (UniqueName: \"kubernetes.io/projected/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-kube-api-access-nlrnd\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.598579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-iptables-alerter-script\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.598579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-modprobe-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.598579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-etc-tuned\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.598579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cni-binary-copy\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.598579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-etc-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.598750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-dbus\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.598750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgv67\" (UniqueName: \"kubernetes.io/projected/cded0e24-b028-447a-9bad-beda2dd20dbb-kube-api-access-zgv67\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.598750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9712dd02-a8c8-402f-bfcd-8e149dc4835b-serviceca\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.598750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-conf-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.598750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-kubelet\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.598750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-host-slash\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.598934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pmk\" (UniqueName: \"kubernetes.io/projected/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-kube-api-access-g7pmk\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.598934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-conf\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.598934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-socket-dir-parent\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.598934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovn-node-metrics-cert\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.598934 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.598881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-kubelet-config\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.599094 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt882\" (UniqueName: \"kubernetes.io/projected/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-kube-api-access-pt882\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:20.599094 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysconfig\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.599094 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-systemd\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.599273 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-var-lib-kubelet\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.599273 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbnk\" (UniqueName: \"kubernetes.io/projected/9712dd02-a8c8-402f-bfcd-8e149dc4835b-kube-api-access-zxbnk\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.599273 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-systemd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.599273 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.599400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-node-log\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.599400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-log-socket\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.599400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-daemon-config\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.599400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-config\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.599400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-env-overrides\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.599555 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.599420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.650011 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.649985 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:28:20.650151 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.650061 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tb9dj" Oct 30 01:28:20.651159 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651123 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:28:20.651294 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651252 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:28:20.651294 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651250 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:28:20.651381 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651364 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:28:20.651413 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651384 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.651597 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651577 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:28:20.651708 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651629 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.651708 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651640 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.651708 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651688 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:28:20.652057 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.651981 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:28:20.657236 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657213 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7lll\"" Oct 30 01:28:20.657340 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657240 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.657580 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657560 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:28:20.657668 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657619 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.657668 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657629 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lzbd2\"" Oct 30 01:28:20.657668 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657658 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:28:20.657788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657668 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lm6wv\"" Oct 30 01:28:20.657788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657667 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.657788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657701 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:28:20.657788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657741 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2gfdm\"" Oct 30 01:28:20.657788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.657763 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:28:20.665678 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.665650 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:28:20.667056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.667042 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglfc\"" Oct 30 01:28:20.667120 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.667059 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:28:20.700267 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-netd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-system-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-systemd-units\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-socket-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2c22\" (UniqueName: \"kubernetes.io/projected/7b706a15-4789-4900-a62b-ba4275904124-kube-api-access-j2c22\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-netd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700423 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-etc-kubernetes\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-bin\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700450 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-etc-kubernetes\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700412 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-system-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-bin\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-systemd-units\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700511 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-socket-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-system-cni-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-device-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-k8s-cni-cncf-io\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700676 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-system-cni-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700695 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-k8s-cni-cncf-io\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700708 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-device-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.700775 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-hostroot\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-registration-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700782 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-hostroot\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-multus-certs\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-registration-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-slash\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-slash\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-script-lib\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-multus-certs\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrnd\" (UniqueName: \"kubernetes.io/projected/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-kube-api-access-nlrnd\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-iptables-alerter-script\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-modprobe-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.700994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-etc-tuned\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cni-binary-copy\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.701464 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-etc-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-dbus\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgv67\" (UniqueName: \"kubernetes.io/projected/cded0e24-b028-447a-9bad-beda2dd20dbb-kube-api-access-zgv67\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701112 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-modprobe-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9712dd02-a8c8-402f-bfcd-8e149dc4835b-serviceca\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-conf-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-kubelet\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-etc-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-host-slash\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pmk\" (UniqueName: \"kubernetes.io/projected/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-kube-api-access-g7pmk\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-conf\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-socket-dir-parent\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovn-node-metrics-cert\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-kubelet-config\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt882\" (UniqueName: \"kubernetes.io/projected/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-kube-api-access-pt882\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysconfig\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-systemd\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701460 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-systemd\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701489 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-conf-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-dbus\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-kubelet\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701535 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-host-slash\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701711 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-conf\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-kubelet-config\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701775 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-socket-dir-parent\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysconfig\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-var-lib-kubelet\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbnk\" (UniqueName: \"kubernetes.io/projected/9712dd02-a8c8-402f-bfcd-8e149dc4835b-kube-api-access-zxbnk\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-systemd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.701996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-node-log\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702020 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-log-socket\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702058 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-systemd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-daemon-config\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.702962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-config\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702112 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-script-lib\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-env-overrides\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702137 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-iptables-alerter-script\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702024 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-var-lib-kubelet\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9712dd02-a8c8-402f-bfcd-8e149dc4835b-serviceca\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-netns\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-multus\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cnibin\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-tmp-dir\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-node-log\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702340 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-kubernetes\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702369 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-run\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-log-socket\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.703832 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702442 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-run\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702441 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702477 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cnibin\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702511 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-netns\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.702532 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-kubernetes\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.702614 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:28:21.202574642 +0000 UTC m=+2.101370574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702634 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-env-overrides\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702752 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-tmp-dir\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-lib-modules\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702833 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-bin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-multus\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-var-lib-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-netns\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pqj\" (UniqueName: \"kubernetes.io/projected/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-kube-api-access-h7pqj\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702930 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-lib-modules\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9x6\" (UniqueName: \"kubernetes.io/projected/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-kube-api-access-jg9x6\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.704746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-var-lib-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702954 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-config\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-bin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.702970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-host\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-netns\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703100 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-host\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-ovn\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703155 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-ovn\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-sys-fs\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-tmp\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9712dd02-a8c8-402f-bfcd-8e149dc4835b-host\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703299 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-sys-fs\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cnibin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703497 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cnibin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5kq\" (UniqueName: \"kubernetes.io/projected/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-kube-api-access-7p5kq\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c859dda-f9c4-4c70-b6d2-e72b990dc465-agent-certs\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.705612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-hosts-file\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-sys\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703257 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703795 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-sys\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703799 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-hosts-file\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703845 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9712dd02-a8c8-402f-bfcd-8e149dc4835b-host\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.703319 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.703906 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:28:21.203889197 +0000 UTC m=+2.102685114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-kubelet\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.703970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c859dda-f9c4-4c70-b6d2-e72b990dc465-konnectivity-ca\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.704251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cni-binary-copy\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.706389 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.704262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-daemon-config\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.704322 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-kubelet\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.704906 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c859dda-f9c4-4c70-b6d2-e72b990dc465-konnectivity-ca\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.704976 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.705436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.706180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-etc-tuned\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.706391 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovn-node-metrics-cert\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.706880 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.706772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c859dda-f9c4-4c70-b6d2-e72b990dc465-agent-certs\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.707146 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.707128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-tmp\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.723451 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.723425 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148264b018741ac927337137ecf951d1.slice/crio-15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018 WatchSource:0}: Error finding container 15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018: Status 404 returned error can't find the container with id 15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018 Oct 30 01:28:20.725399 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.725378 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc72893a455f4d88649494269f87258.slice/crio-766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11 WatchSource:0}: Error finding container 766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11: Status 404 returned error can't find the container with id 766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11 Oct 30 01:28:20.730124 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.730109 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:28:20.734342 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.734319 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2c22\" (UniqueName: \"kubernetes.io/projected/7b706a15-4789-4900-a62b-ba4275904124-kube-api-access-j2c22\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.734469 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.734449 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrnd\" (UniqueName: \"kubernetes.io/projected/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-kube-api-access-nlrnd\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.737594 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.737570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt882\" (UniqueName: \"kubernetes.io/projected/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-kube-api-access-pt882\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:20.737986 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.737963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgv67\" (UniqueName: \"kubernetes.io/projected/cded0e24-b028-447a-9bad-beda2dd20dbb-kube-api-access-zgv67\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.738064 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.738014 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:20.738064 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.738034 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:20.738064 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.738045 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:20.738180 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:20.738092 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:21.238077953 +0000 UTC m=+2.136873871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:20.739358 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.739338 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbnk\" (UniqueName: \"kubernetes.io/projected/9712dd02-a8c8-402f-bfcd-8e149dc4835b-kube-api-access-zxbnk\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.742997 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.742956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9x6\" (UniqueName: \"kubernetes.io/projected/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-kube-api-access-jg9x6\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.743084 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.743021 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pmk\" (UniqueName: \"kubernetes.io/projected/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-kube-api-access-g7pmk\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.743084 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.743046 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pqj\" (UniqueName: \"kubernetes.io/projected/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-kube-api-access-h7pqj\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.744925 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.744907 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5kq\" (UniqueName: \"kubernetes.io/projected/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-kube-api-access-7p5kq\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.855799 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.855766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:28:20.866540 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.866517 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcded0e24_b028_447a_9bad_beda2dd20dbb.slice/crio-b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016 WatchSource:0}: Error finding container b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016: Status 404 returned error can't find the container with id b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016 Oct 30 01:28:20.878057 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.878034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:28:20.887114 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.887088 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b706a15_4789_4900_a62b_ba4275904124.slice/crio-0dcd02fa1874df04b21baf9b6f1b806107ea51160d1ea2f5aa6d76e9ede717c1 WatchSource:0}: Error finding container 0dcd02fa1874df04b21baf9b6f1b806107ea51160d1ea2f5aa6d76e9ede717c1: Status 404 returned error can't find the container with id 0dcd02fa1874df04b21baf9b6f1b806107ea51160d1ea2f5aa6d76e9ede717c1 Oct 30 01:28:20.902720 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.902699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:28:20.905056 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.905038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gcqqc" Oct 30 01:28:20.913454 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.913430 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9712dd02_a8c8_402f_bfcd_8e149dc4835b.slice/crio-e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e WatchSource:0}: Error finding container e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e: Status 404 returned error can't find the container with id e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e Oct 30 01:28:20.916169 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.916146 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6eff6e_ebcf_45c0_a37f_3cd99bc787e9.slice/crio-d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d WatchSource:0}: Error finding container d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d: Status 404 returned error can't find the container with id d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d Oct 30 01:28:20.927841 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.927821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:20.937598 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.937571 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c859dda_f9c4_4c70_b6d2_e72b990dc465.slice/crio-990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c WatchSource:0}: Error finding container 990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c: Status 404 returned error can't find the container with id 990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c Oct 30 01:28:20.942751 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.942731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:28:20.951872 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.951849 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467074bb_b3a4_444d_a8b9_9aae4f9d9d56.slice/crio-ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c WatchSource:0}: Error finding container ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c: Status 404 returned error can't find the container with id ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c Oct 30 01:28:20.978301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.978281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:20.987604 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.987582 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0245e1_cf5c_4d84_8072_bdd72bfe3f36.slice/crio-1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9 WatchSource:0}: Error finding container 1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9: Status 404 returned error can't find the container with id 1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9 Oct 30 01:28:20.989702 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.989657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5w55t" Oct 30 01:28:20.997269 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:20.997251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:28:20.998752 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:20.998731 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b9ca87_bcc5_4b7a_8517_db3484b7c5c8.slice/crio-be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77 WatchSource:0}: Error finding container be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77: Status 404 returned error can't find the container with id be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77 Oct 30 01:28:21.005489 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:28:21.005465 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14d41c9_2491_4d02_83aa_356a6e8aa1e5.slice/crio-fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833 WatchSource:0}: Error finding container fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833: Status 404 returned error can't find the container with id fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833 Oct 30 01:28:21.207827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.207257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:21.207827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.207312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:21.207827 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.207472 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:21.207827 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.207488 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:21.207827 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.207522 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:28:22.207508073 +0000 UTC m=+3.106303996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:21.207827 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.207546 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:28:22.207531348 +0000 UTC m=+3.106327271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:21.308499 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.308453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:21.308727 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.308633 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:21.308727 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.308657 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:21.308727 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.308671 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:21.308915 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:21.308737 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:22.308714974 +0000 UTC m=+3.207510913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:21.509773 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.509697 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:21.653349 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.653281 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:21 +0000 UTC" deadline="2028-04-02 09:02:59.166586033 +0000 UTC" Oct 30 01:28:21.653349 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.653343 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12487h34m37.513247097s" Oct 30 01:28:21.655733 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.655668 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7c4wq" event={"ID":"467074bb-b3a4-444d-a8b9-9aae4f9d9d56","Type":"ContainerStarted","Data":"ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c"} Oct 30 01:28:21.662087 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.661993 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lwf9z" event={"ID":"9c859dda-f9c4-4c70-b6d2-e72b990dc465","Type":"ContainerStarted","Data":"990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c"} Oct 30 01:28:21.666161 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.666077 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" event={"ID":"148264b018741ac927337137ecf951d1","Type":"ContainerStarted","Data":"15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018"} Oct 30 01:28:21.678400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.678354 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerStarted","Data":"fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833"} Oct 30 01:28:21.696945 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.696910 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gcqqc" event={"ID":"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9","Type":"ContainerStarted","Data":"d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d"} Oct 30 01:28:21.707604 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.707564 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qmg28" event={"ID":"9712dd02-a8c8-402f-bfcd-8e149dc4835b","Type":"ContainerStarted","Data":"e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e"} Oct 30 01:28:21.711690 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.711659 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s98cl" event={"ID":"7b706a15-4789-4900-a62b-ba4275904124","Type":"ContainerStarted","Data":"0dcd02fa1874df04b21baf9b6f1b806107ea51160d1ea2f5aa6d76e9ede717c1"} Oct 30 01:28:21.718278 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.718246 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" event={"ID":"cded0e24-b028-447a-9bad-beda2dd20dbb","Type":"ContainerStarted","Data":"b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016"} Oct 30 01:28:21.728363 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.728327 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" event={"ID":"0fc72893a455f4d88649494269f87258","Type":"ContainerStarted","Data":"766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11"} Oct 30 01:28:21.741089 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.741050 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5w55t" event={"ID":"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8","Type":"ContainerStarted","Data":"be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77"} Oct 30 01:28:21.745861 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.745827 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9"} Oct 30 01:28:21.898069 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.897943 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:21.951800 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:21.951739 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:22.216993 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.216911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:22.216993 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.216967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:22.217328 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.217309 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:22.217399 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.217384 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:28:24.217363769 +0000 UTC m=+5.116159702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:22.217811 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.217789 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:22.217897 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.217849 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:28:24.217834728 +0000 UTC m=+5.116630653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:22.318079 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.317421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:22.318079 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.317625 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:22.318079 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.317645 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:22.318079 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.317660 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:22.318079 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.317718 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:24.317700385 +0000 UTC m=+5.216496318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:22.629949 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.629861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:22.630475 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.630025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:22.630552 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.630512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:22.630640 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.630620 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:22.630727 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.630710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:22.630813 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:22.630793 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:22.654202 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.654161 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:21 +0000 UTC" deadline="2028-04-09 02:34:41.546987063 +0000 UTC" Oct 30 01:28:22.654202 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:22.654200 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12649h6m18.892791885s" Oct 30 01:28:24.234964 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:24.234921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:24.235480 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:24.234986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:24.235480 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.235184 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:24.235480 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.235259 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:28:28.235238943 +0000 UTC m=+9.134034862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:24.235480 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.235314 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:24.235480 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.235443 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:28:28.235422415 +0000 UTC m=+9.134218351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:24.336029 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:24.335440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:24.336029 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.335607 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:24.336029 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.335626 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:24.336029 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.335640 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:24.336029 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.335693 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:28.335679267 +0000 UTC m=+9.234475190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:24.629773 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:24.629730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:24.629950 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.629896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:24.630001 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:24.629978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:24.630111 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.630056 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:24.630205 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:24.630120 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:24.630271 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:24.630217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:25.790541 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.789432 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" event={"ID":"0fc72893a455f4d88649494269f87258","Type":"ContainerStarted","Data":"b327a9025af5d9a64688ec3ef42ce7ca8b991e7a6726a2f9ab290170e3dcc499"} Oct 30 01:28:25.797019 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.796985 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lwf9z" event={"ID":"9c859dda-f9c4-4c70-b6d2-e72b990dc465","Type":"ContainerStarted","Data":"be3fb531323783d04b15e2ada0d17912957e6d7a1bdb07e894f0595e43d82c92"} Oct 30 01:28:25.801538 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.801503 2577 generic.go:356] "Generic (PLEG): container finished" podID="148264b018741ac927337137ecf951d1" containerID="02eeddbc0faf83a6cec7d089651936247a2b5806955bef4f8fb4dfe140fe2338" exitCode=0 Oct 30 01:28:25.801660 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.801592 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" event={"ID":"148264b018741ac927337137ecf951d1","Type":"ContainerDied","Data":"02eeddbc0faf83a6cec7d089651936247a2b5806955bef4f8fb4dfe140fe2338"} Oct 30 01:28:25.812001 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.811967 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="1241fe39afd5adfbc3a1b19129861c2661ddc432a76cbe25bd199dda9dd6c9dc" exitCode=0 Oct 30 01:28:25.812113 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.812019 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerDied","Data":"1241fe39afd5adfbc3a1b19129861c2661ddc432a76cbe25bd199dda9dd6c9dc"} Oct 30 01:28:25.845531 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:25.845442 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" podStartSLOduration=5.845418817 podStartE2EDuration="5.845418817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:25.84308685 +0000 UTC m=+6.741882793" watchObservedRunningTime="2026-10-30 01:28:25.845418817 +0000 UTC m=+6.744214761" Oct 30 01:28:26.630575 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:26.630538 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:26.630761 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:26.630691 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:26.631155 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:26.631124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:26.631294 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:26.631253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:26.631359 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:26.631324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:26.631425 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:26.631404 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:26.825858 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:26.825823 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" event={"ID":"148264b018741ac927337137ecf951d1","Type":"ContainerStarted","Data":"ddf48252da9806db38e38ca18513d49203be5ed3c21396657c454fdba2212e85"} Oct 30 01:28:26.874400 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:26.874335 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lwf9z" podStartSLOduration=2.300465727 podStartE2EDuration="6.87431101s" totalImagesPullingTime="4.573845283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:25.998304774 +0000 UTC m=+6.897100717" watchObservedRunningTime="2026-10-30 01:28:26.87431101 +0000 UTC m=+7.773106953" Oct 30 01:28:26.874725 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:26.874689 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" podStartSLOduration=6.874678319 podStartE2EDuration="6.874678319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:26.873418743 +0000 UTC m=+7.772214692" watchObservedRunningTime="2026-10-30 01:28:26.874678319 +0000 UTC m=+7.773474262" Oct 30 01:28:27.829854 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:27.829404 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qmg28" event={"ID":"9712dd02-a8c8-402f-bfcd-8e149dc4835b","Type":"ContainerStarted","Data":"0dfabfb569fc65da6345d0b5c4ec808641c11e114d938cc3b56ddb8459fcebb6"} Oct 30 01:28:28.280853 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.280817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:28.281022 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.280869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:28.281022 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.280997 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:28.281154 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.281053 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:28:36.281035242 +0000 UTC m=+17.179831162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:28.281488 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.281461 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:28.281556 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.281519 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:28:36.281503837 +0000 UTC m=+17.180299768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:28.382057 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.381970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:28.382215 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.382136 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:28.382215 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.382155 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:28.382215 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.382167 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:28.382360 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.382224 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:36.382206502 +0000 UTC m=+17.281002424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:28.629798 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.629753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:28.629952 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.629911 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:28.630577 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.630359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:28.630577 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.630389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:28.630577 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.630467 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:28.630577 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:28.630540 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:28.838328 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:28.838183 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" event={"ID":"cded0e24-b028-447a-9bad-beda2dd20dbb","Type":"ContainerStarted","Data":"8d893ad7d07224b8a3ae1895776d28a80bcc3c3379951618d7217cc145ddb0d8"} Oct 30 01:28:29.402833 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.402785 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:28:29.564453 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.564311 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:28:29.402801153Z","UUID":"4b661983-e191-4a96-b7ab-762bdf019ebd","Handler":null,"Name":"","Endpoint":""} Oct 30 01:28:29.570291 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.570268 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:28:29.570291 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.570295 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:28:29.842696 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.842621 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gcqqc" event={"ID":"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9","Type":"ContainerStarted","Data":"dde6918b35a777051a40d9395463fcfe85133e57211d0c3413c70a29f0c67ab0"} Oct 30 01:28:29.848344 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.848279 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" event={"ID":"cded0e24-b028-447a-9bad-beda2dd20dbb","Type":"ContainerStarted","Data":"7b5f822084e7ec58a7bcc74c932750cf96955f91f9241bcd9672e2f78d811060"} Oct 30 01:28:29.853330 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.852217 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5w55t" event={"ID":"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8","Type":"ContainerStarted","Data":"b40be6adb78007348834ac5423528778f3779949a9d7f0f9521a5a04ae2e9e92"} Oct 30 01:28:29.896785 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.896706 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qmg28" podStartSLOduration=3.902694651 podStartE2EDuration="9.896686042s" totalImagesPullingTime="5.993991391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:27.87397914 +0000 UTC m=+8.772775077" watchObservedRunningTime="2026-10-30 01:28:29.896686042 +0000 UTC m=+10.795481983" Oct 30 01:28:29.896939 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:29.896811 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gcqqc" podStartSLOduration=1.576770154 podStartE2EDuration="9.896802922s" totalImagesPullingTime="8.320032768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:29.895854244 +0000 UTC m=+10.794650184" watchObservedRunningTime="2026-10-30 01:28:29.896802922 +0000 UTC m=+10.795598863" Oct 30 01:28:30.254043 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.248559 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:30.254043 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.249406 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:30.315798 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.314740 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5w55t" podStartSLOduration=1.865322866 podStartE2EDuration="10.314717317s" totalImagesPullingTime="8.449394451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:29.954885191 +0000 UTC m=+10.853681131" watchObservedRunningTime="2026-10-30 01:28:30.314717317 +0000 UTC m=+11.213513257" Oct 30 01:28:30.629926 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.629890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:30.629926 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.629922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:30.630144 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.629922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:30.630144 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:30.630026 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:30.630144 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:30.630085 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:30.630299 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:30.630176 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:30.856926 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.856609 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="eca6775891be01b1d265d26a39a80822296015fef82f27162503ba29e256750b" exitCode=0 Oct 30 01:28:30.856926 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.856702 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerDied","Data":"eca6775891be01b1d265d26a39a80822296015fef82f27162503ba29e256750b"} Oct 30 01:28:30.861195 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.861167 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" event={"ID":"cded0e24-b028-447a-9bad-beda2dd20dbb","Type":"ContainerStarted","Data":"f7842208b763151090a39c24615bf4cfae1d8f870452e500467742ac95d6eb0b"} Oct 30 01:28:30.862301 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.862278 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:30.862376 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:30.862342 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:28:31.009496 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.009388 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" podStartSLOduration=1.981185348 podStartE2EDuration="11.009368395s" totalImagesPullingTime="9.028183047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:30.959016675 +0000 UTC m=+11.857812615" watchObservedRunningTime="2026-10-30 01:28:31.009368395 +0000 UTC m=+11.908164336" Oct 30 01:28:31.865527 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.865440 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7c4wq" event={"ID":"467074bb-b3a4-444d-a8b9-9aae4f9d9d56","Type":"ContainerStarted","Data":"cdd614c5fbb556d04d87b3a772acd69b3ec506613ac65c5519bf453660323c6f"} Oct 30 01:28:31.867954 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.867917 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="1d9fcb617733900d422a8cd1dde865286b6189e612ee7e5c31465180056f017d" exitCode=0 Oct 30 01:28:31.868067 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.868006 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerDied","Data":"1d9fcb617733900d422a8cd1dde865286b6189e612ee7e5c31465180056f017d"} Oct 30 01:28:31.869795 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.869771 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s98cl" event={"ID":"7b706a15-4789-4900-a62b-ba4275904124","Type":"ContainerStarted","Data":"82de0a0f1b6ac9c4555187104c7168959818fe83c20c5ba233c8808427714774"} Oct 30 01:28:31.960710 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.960658 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7c4wq" podStartSLOduration=2.984820545 podStartE2EDuration="11.960644748s" totalImagesPullingTime="8.975824203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:31.910995812 +0000 UTC m=+12.809791751" watchObservedRunningTime="2026-10-30 01:28:31.960644748 +0000 UTC m=+12.859440687" Oct 30 01:28:31.961345 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:31.961260 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s98cl" podStartSLOduration=1.290804195 podStartE2EDuration="11.961249325s" totalImagesPullingTime="10.67044513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:31.960802751 +0000 UTC m=+12.859598683" watchObservedRunningTime="2026-10-30 01:28:31.961249325 +0000 UTC m=+12.860045281" Oct 30 01:28:32.630514 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:32.630488 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:32.630651 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:32.630630 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:32.630707 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:32.630488 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:32.630760 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:32.630718 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:32.630760 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:32.630488 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:32.630825 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:32.630787 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:32.875176 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:32.875086 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="8fcb2e618fd648d310613e81229f83571eb1dfa23ec63ed68fb834475e590aa3" exitCode=0 Oct 30 01:28:32.875176 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:32.875163 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerDied","Data":"8fcb2e618fd648d310613e81229f83571eb1dfa23ec63ed68fb834475e590aa3"} Oct 30 01:28:34.630051 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:34.630011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:34.630051 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:34.630038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:34.630750 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:34.630020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:34.630750 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:34.630164 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:34.630750 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:34.630272 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:34.630750 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:34.630357 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:34.882630 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:34.882600 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerStarted","Data":"f8a02ac79be0f93f88c4b03b8ae462b83a349c2678440de10915fe1f49c2762a"} Oct 30 01:28:35.888006 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:35.887806 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="f8a02ac79be0f93f88c4b03b8ae462b83a349c2678440de10915fe1f49c2762a" exitCode=0 Oct 30 01:28:35.888006 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:35.887891 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerDied","Data":"f8a02ac79be0f93f88c4b03b8ae462b83a349c2678440de10915fe1f49c2762a"} Oct 30 01:28:36.345659 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.345634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:36.345788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.345667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:36.345788 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.345776 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:36.345788 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.345778 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:36.345942 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.345819 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:28:52.345807719 +0000 UTC m=+33.244603635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:36.345942 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.345833 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:28:52.34582626 +0000 UTC m=+33.244622176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:36.446958 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.446930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:36.447055 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.447042 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:36.447099 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.447064 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:36.447099 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.447074 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:36.447263 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.447110 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:52.447097026 +0000 UTC m=+33.345892954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:36.629859 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.629825 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:36.629859 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.629848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:36.629859 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.629833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:36.630058 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.629942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:36.630058 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.629987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:36.630144 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:36.630070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:36.893028 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.892937 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="d50286d30562bbd539a955ddc5fa27c633ad0d574ab950dde1eae08198759f1b" exitCode=0 Oct 30 01:28:36.893514 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.893023 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerDied","Data":"d50286d30562bbd539a955ddc5fa27c633ad0d574ab950dde1eae08198759f1b"} Oct 30 01:28:36.895939 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.895924 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:28:36.896251 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896230 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" containerID="bfd791706e80bdc402c72e5544ec8e8c95afc96df2e52f43dab6157b59142873" exitCode=1 Oct 30 01:28:36.896308 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896269 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"a9cbb87ca4b88f1b4e45ad784aa524f6dfb9d73f8d8daba94c03406f5ef55e67"} Oct 30 01:28:36.896350 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"15423c7ae678a2d7d165169f481db432039621f98f31377c796ec0e840993554"} Oct 30 01:28:36.896378 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896350 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"f17eb4bc8f770ec63635d4a3ab9980a3412d855d3ec4691b0925a87feacf38b5"} Oct 30 01:28:36.896412 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896379 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"22cb45c17de521bd4ab67f0498a0f3134ee3c835f18d348c16cde84ea7ac9f5d"} Oct 30 01:28:36.896412 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896408 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerDied","Data":"bfd791706e80bdc402c72e5544ec8e8c95afc96df2e52f43dab6157b59142873"} Oct 30 01:28:36.896469 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:36.896434 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"cde49033cbf8cb7f87c6bfee54902162a5c841e2f72726da3f65e9c910a98f93"} Oct 30 01:28:37.901310 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:37.901273 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" event={"ID":"d14d41c9-2491-4d02-83aa-356a6e8aa1e5","Type":"ContainerStarted","Data":"0b7c78c54e2d1573bed8ad4caff6ac2e0b61f22e4058ee6853368b3547762324"} Oct 30 01:28:37.981311 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:37.981256 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f9pb2" podStartSLOduration=7.461210461 podStartE2EDuration="17.981242791s" totalImagesPullingTime="10.52003233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:37.980693862 +0000 UTC m=+18.879489800" watchObservedRunningTime="2026-10-30 01:28:37.981242791 +0000 UTC m=+18.880038728" Oct 30 01:28:38.629890 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:38.629859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:38.630044 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:38.629929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:38.630190 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:38.630050 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:38.630190 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:38.630098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:38.630311 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:38.630187 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:38.630311 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:38.630244 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:38.906183 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:38.906107 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:28:38.906554 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:38.906448 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"af1afbe2881ed53f67809842b06c2ac1887ab2c89fac4ef8bd079c2486eb7639"} Oct 30 01:28:39.076183 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:39.076156 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5w55t_58b9ca87-bcc5-4b7a-8517-db3484b7c5c8/dns-node-resolver/0.log" Oct 30 01:28:40.409754 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.409725 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmg28_9712dd02-a8c8-402f-bfcd-8e149dc4835b/node-ca/0.log" Oct 30 01:28:40.629884 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.629863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:40.629977 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.629921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:40.630030 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.630003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:40.630030 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:40.630018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:40.630151 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:40.630088 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:40.630203 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:40.630144 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:40.913226 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.913198 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:28:40.913581 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.913556 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"b57d878f4fa1fae7513594aac8e9eae03f91bdecddf1c3b81ea458e744e80960"} Oct 30 01:28:40.913802 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.913779 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:40.913930 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.913848 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:40.913995 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.913942 2577 scope.go:122] "RemoveContainer" containerID="bfd791706e80bdc402c72e5544ec8e8c95afc96df2e52f43dab6157b59142873" Oct 30 01:28:40.929036 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.929017 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:40.929409 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:40.929393 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:41.918954 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:41.918781 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:28:41.919639 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:41.919323 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" event={"ID":"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36","Type":"ContainerStarted","Data":"e41702e0c96a3b692762afe19c02391f525deca58429d4d41dc62c5df7c5269c"} Oct 30 01:28:41.919639 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:41.919386 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:41.993085 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:41.993031 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" podStartSLOduration=7.07726426 podStartE2EDuration="21.993018236s" totalImagesPullingTime="14.915753976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:41.992639506 +0000 UTC m=+22.891435445" watchObservedRunningTime="2026-10-30 01:28:41.993018236 +0000 UTC m=+22.891814174" Oct 30 01:28:42.210202 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:42.210133 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:28:42.630138 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:42.630090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:42.630138 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:42.630109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:42.630343 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:42.630145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:42.630343 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:42.630218 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:42.630343 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:42.630254 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:42.630445 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:42.630347 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:44.629941 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:44.629901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:44.630469 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:44.629911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:44.630469 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:44.630016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:44.630469 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:44.629911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:44.630469 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:44.630088 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:44.630469 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:44.630182 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:46.629837 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:46.629799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:46.629837 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:46.629826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:46.630334 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:46.629915 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:46.630334 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:46.629946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:46.630334 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:46.630010 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:46.630334 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:46.630067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:48.630327 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:48.630290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:48.630327 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:48.630313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:48.630769 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:48.630387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:48.630769 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:48.630403 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:48.630769 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:48.630459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:48.630769 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:48.630523 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:50.629699 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:50.629665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:50.630186 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:50.629727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:50.630186 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:50.629820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:50.630186 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:50.629842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:50.630186 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:50.629960 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:50.630186 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:50.630039 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:52.355579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:52.355552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:52.355962 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:52.355604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:52.355962 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.355686 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:52.355962 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.355690 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:52.355962 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.355737 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs podName:ba4999ae-bde0-4c8a-9652-4ea7811afdaa nodeName:}" failed. No retries permitted until 2026-10-30 01:29:24.355720512 +0000 UTC m=+65.254516448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs") pod "network-metrics-daemon-xv85r" (UID: "ba4999ae-bde0-4c8a-9652-4ea7811afdaa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:52.355962 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.355750 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret podName:141993bb-335c-4e9c-b08e-da41dd9d7f0f nodeName:}" failed. No retries permitted until 2026-10-30 01:29:24.35574447 +0000 UTC m=+65.254540386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret") pod "global-pull-secret-syncer-jt6kw" (UID: "141993bb-335c-4e9c-b08e-da41dd9d7f0f") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:52.455885 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:52.455856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:52.456038 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.455986 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:52.456038 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.456003 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:52.456038 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.456016 2577 projected.go:196] Error preparing data for projected volume kube-api-access-sgf5g for pod openshift-network-diagnostics/network-check-target-9qlzc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:52.456186 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.456073 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g podName:131e7807-d6f8-4104-836e-33c6dcbbc286 nodeName:}" failed. No retries permitted until 2026-10-30 01:29:24.45605558 +0000 UTC m=+65.354851512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgf5g" (UniqueName: "kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g") pod "network-check-target-9qlzc" (UID: "131e7807-d6f8-4104-836e-33c6dcbbc286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:52.630625 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:52.630555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:52.630790 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:52.630566 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:52.630790 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.630664 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:52.630790 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:52.630566 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:52.630790 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.630736 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:52.630932 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:52.630788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:54.630624 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:54.630592 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:54.630624 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:54.630624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:54.631062 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:54.630697 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:54.631062 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:54.630738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:54.631062 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:54.630832 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:54.631062 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:54.630921 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:56.630610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:56.630578 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:56.630610 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:56.630609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:56.631007 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:56.630668 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:28:56.631007 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:56.630708 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:56.631007 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:56.630813 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:56.631007 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:56.630893 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:58.630393 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:58.630357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:28:58.630393 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:58.630389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:28:58.630791 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:58.630455 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:28:58.630791 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:58.630553 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:28:58.630791 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:28:58.630608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:28:58.630791 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:28:58.630653 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:29:00.629861 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.629822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:00.630394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.629839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:00.630394 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:00.629950 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:29:00.630394 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.629954 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:00.630394 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:00.630046 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:29:00.630394 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:00.630167 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:29:00.799072 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.799044 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jt6kw"] Oct 30 01:29:00.801095 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.801067 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9qlzc"] Oct 30 01:29:00.804217 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.804154 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xv85r"] Oct 30 01:29:00.958063 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.958038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:00.958243 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.958038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:00.958243 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:00.958161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:29:00.958243 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:00.958038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:00.958388 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:00.958249 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:29:00.958388 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:00.958285 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:29:01.543761 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:01.543701 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5w55t_58b9ca87-bcc5-4b7a-8517-db3484b7c5c8/dns-node-resolver/0.log" Oct 30 01:29:02.630534 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:02.630509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:02.630980 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:02.630508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:02.630980 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:02.630602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:29:02.630980 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:02.630508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:02.630980 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:02.630693 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:29:02.630980 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:02.630788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:29:03.144192 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:03.143947 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmg28_9712dd02-a8c8-402f-bfcd-8e149dc4835b/node-ca/0.log" Oct 30 01:29:04.629980 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.629953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:04.630361 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.629989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:04.630361 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.630045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:04.630361 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:04.630063 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv85r" podUID="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" Oct 30 01:29:04.630361 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:04.630142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jt6kw" podUID="141993bb-335c-4e9c-b08e-da41dd9d7f0f" Oct 30 01:29:04.630361 ip-10-0-143-27 kubenswrapper[2577]: E1030 01:29:04.630220 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9qlzc" podUID="131e7807-d6f8-4104-836e-33c6dcbbc286" Oct 30 01:29:04.845098 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.845079 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeReady" Oct 30 01:29:04.845209 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.845200 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 01:29:04.912215 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.912141 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gm2cs"] Oct 30 01:29:04.914231 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.914207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:04.919079 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.919052 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zdlwv"] Oct 30 01:29:04.921156 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.921117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:04.933417 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.933400 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:29:04.933553 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.933539 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zqs4x\"" Oct 30 01:29:04.938039 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.938022 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:29:04.938100 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.938046 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:29:04.940286 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f2c416d-69b2-49ef-a2e3-230c5935d67b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:04.940348 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f2c416d-69b2-49ef-a2e3-230c5935d67b-data-volume\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:04.940348 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbb8q\" (UniqueName: \"kubernetes.io/projected/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-api-access-cbb8q\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:04.940449 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d69ca43-18c0-4010-87a9-7e3473dfe936-cert\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:04.940483 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbr9\" (UniqueName: \"kubernetes.io/projected/9d69ca43-18c0-4010-87a9-7e3473dfe936-kube-api-access-8sbr9\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:04.940513 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f2c416d-69b2-49ef-a2e3-230c5935d67b-crio-socket\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:04.940544 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.940525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:04.942788 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.942769 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gm2cs"] Oct 30 01:29:04.947054 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:04.947035 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdlwv"] Oct 30 01:29:05.014931 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.014900 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9gx4n"] Oct 30 01:29:05.016627 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.016612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.034065 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.034041 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfcpg\"" Oct 30 01:29:05.034065 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.034048 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:29:05.034898 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.034882 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2p848\"" Oct 30 01:29:05.034898 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.034882 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:29:05.038827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.038805 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:29:05.039038 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.039021 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:29:05.039038 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.039035 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:29:05.039145 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.039053 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:29:05.041054 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041033 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9gx4n"] Oct 30 01:29:05.041212 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f2c416d-69b2-49ef-a2e3-230c5935d67b-crio-socket\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.041271 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.041271 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f2c416d-69b2-49ef-a2e3-230c5935d67b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.041377 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f2c416d-69b2-49ef-a2e3-230c5935d67b-data-volume\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.041377 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbb8q\" (UniqueName: \"kubernetes.io/projected/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-api-access-cbb8q\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.041377 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-metrics-tls\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.041523 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gmk\" (UniqueName: \"kubernetes.io/projected/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-kube-api-access-n8gmk\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.041523 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-tmp-dir\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.041523 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d69ca43-18c0-4010-87a9-7e3473dfe936-cert\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:05.041523 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbr9\" (UniqueName: \"kubernetes.io/projected/9d69ca43-18c0-4010-87a9-7e3473dfe936-kube-api-access-8sbr9\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:05.041523 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f2c416d-69b2-49ef-a2e3-230c5935d67b-crio-socket\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.041758 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-config-volume\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.042006 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.041983 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f2c416d-69b2-49ef-a2e3-230c5935d67b-data-volume\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.042187 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.042164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.045776 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.045755 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f2c416d-69b2-49ef-a2e3-230c5935d67b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.045871 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.045811 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d69ca43-18c0-4010-87a9-7e3473dfe936-cert\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:05.070272 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.070248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbb8q\" (UniqueName: \"kubernetes.io/projected/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-api-access-cbb8q\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.070754 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.070733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbr9\" (UniqueName: \"kubernetes.io/projected/9d69ca43-18c0-4010-87a9-7e3473dfe936-kube-api-access-8sbr9\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:05.142416 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.142383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-metrics-tls\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.142416 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.142415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gmk\" (UniqueName: \"kubernetes.io/projected/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-kube-api-access-n8gmk\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.142629 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.142436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-tmp-dir\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.142629 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.142570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-config-volume\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.142744 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.142736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-tmp-dir\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.143018 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.142999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-config-volume\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.144626 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.144607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-metrics-tls\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.167696 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.167651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gmk\" (UniqueName: \"kubernetes.io/projected/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-kube-api-access-n8gmk\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.225237 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.225202 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:05.233020 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.233005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:05.327515 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.327480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:05.535875 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.535852 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gm2cs"] Oct 30 01:29:05.538696 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:05.538667 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d69ca43_18c0_4010_87a9_7e3473dfe936.slice/crio-67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7 WatchSource:0}: Error finding container 67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7: Status 404 returned error can't find the container with id 67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7 Oct 30 01:29:05.550727 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.550691 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdlwv"] Oct 30 01:29:05.553754 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:05.553725 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2c416d_69b2_49ef_a2e3_230c5935d67b.slice/crio-22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627 WatchSource:0}: Error finding container 22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627: Status 404 returned error can't find the container with id 22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627 Oct 30 01:29:05.621851 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.621829 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9gx4n"] Oct 30 01:29:05.624679 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:05.624654 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0bb2dd_c6e3_430c_b9a2_a00b0f4a4603.slice/crio-6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa WatchSource:0}: Error finding container 6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa: Status 404 returned error can't find the container with id 6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa Oct 30 01:29:05.970155 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.970052 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gx4n" event={"ID":"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603","Type":"ContainerStarted","Data":"6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa"} Oct 30 01:29:05.971826 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.971798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdlwv" event={"ID":"7f2c416d-69b2-49ef-a2e3-230c5935d67b","Type":"ContainerStarted","Data":"7719ccf17e90279153dbccfe1cb9bda225e422a0543896205f959b2882e1a56b"} Oct 30 01:29:05.971940 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.971853 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdlwv" event={"ID":"7f2c416d-69b2-49ef-a2e3-230c5935d67b","Type":"ContainerStarted","Data":"22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627"} Oct 30 01:29:05.973048 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:05.972995 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gm2cs" event={"ID":"9d69ca43-18c0-4010-87a9-7e3473dfe936","Type":"ContainerStarted","Data":"67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7"} Oct 30 01:29:06.630407 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.630367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:06.630407 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.630388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:06.630626 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.630416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:06.648011 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.647989 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:29:06.662494 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.662449 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:29:06.662494 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.662489 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:29:06.662736 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.662625 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:29:06.663476 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.663357 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:29:06.663476 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.663419 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:29:06.676824 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.676776 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:29:06.979265 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:06.979182 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdlwv" event={"ID":"7f2c416d-69b2-49ef-a2e3-230c5935d67b","Type":"ContainerStarted","Data":"ef52d38efbe165e60071faa60b7edef1c332cb269924946ab25eba2dc386d4af"} Oct 30 01:29:07.500086 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.498920 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf"] Oct 30 01:29:07.502896 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.502875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.513554 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.513531 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8mp4k"] Oct 30 01:29:07.515816 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.515796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.527939 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.527917 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf"] Oct 30 01:29:07.560892 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.560869 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-metrics-client-ca\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561016 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.560902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e07d2c61-a1ab-4ece-8205-6515c8b8d880-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.561016 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.560930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-root\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561016 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.560945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-wtmp\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561016 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.560962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-tls\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561016 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.560982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.561016 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-textfile\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjk5\" (UniqueName: \"kubernetes.io/projected/76fcebb0-d866-476d-859e-f8b8626a4a79-kube-api-access-2mjk5\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-sys\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.561403 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.561186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hf5v\" (UniqueName: \"kubernetes.io/projected/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-api-access-7hf5v\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.568465 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.568001 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:29:07.568465 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.568220 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:29:07.568465 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.568338 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tntm7\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570449 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570536 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f9jp6\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570582 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570643 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570784 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570841 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.570941 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:29:07.571060 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.571007 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-metrics-client-ca\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e07d2c61-a1ab-4ece-8205-6515c8b8d880-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-root\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-wtmp\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-tls\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-textfile\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjk5\" (UniqueName: \"kubernetes.io/projected/76fcebb0-d866-476d-859e-f8b8626a4a79-kube-api-access-2mjk5\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-sys\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hf5v\" (UniqueName: \"kubernetes.io/projected/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-api-access-7hf5v\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.663316 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.662935 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-metrics-client-ca\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.664226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.664852 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-sys\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.664966 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-wtmp\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.665011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.665237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e07d2c61-a1ab-4ece-8205-6515c8b8d880-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.665288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-root\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.665283 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.666150 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.665601 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-textfile\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.670164 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.667974 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.671698 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.671396 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-tls\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.672474 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.672418 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.674320 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.674296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.691588 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.691527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjk5\" (UniqueName: \"kubernetes.io/projected/76fcebb0-d866-476d-859e-f8b8626a4a79-kube-api-access-2mjk5\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.693227 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.693196 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hf5v\" (UniqueName: \"kubernetes.io/projected/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-api-access-7hf5v\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.842497 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.842470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:07.847342 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.847319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:07.856300 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:07.856269 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fcebb0_d866_476d_859e_f8b8626a4a79.slice/crio-2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd WatchSource:0}: Error finding container 2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd: Status 404 returned error can't find the container with id 2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd Oct 30 01:29:07.984285 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.984202 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gx4n" event={"ID":"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603","Type":"ContainerStarted","Data":"4e1a9bd34a6046574c322ae5abb379e092dc4f488e129a92f4bf4410866f070b"} Oct 30 01:29:07.984724 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.984309 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gx4n" event={"ID":"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603","Type":"ContainerStarted","Data":"4e6b957624bb867d1ee57e37d634a3656a2ee21ba002072ff00699c47d915240"} Oct 30 01:29:07.984724 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.984342 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:07.985691 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.985664 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mp4k" event={"ID":"76fcebb0-d866-476d-859e-f8b8626a4a79","Type":"ContainerStarted","Data":"2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd"} Oct 30 01:29:07.988096 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:07.988070 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdlwv" event={"ID":"7f2c416d-69b2-49ef-a2e3-230c5935d67b","Type":"ContainerStarted","Data":"32a01e71c6374911a4b84d4ab2187c02b285645db688376b7f353d8b2e30aa85"} Oct 30 01:29:08.032674 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.032613 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9gx4n" podStartSLOduration=2.21764977 podStartE2EDuration="4.032597061s" totalImagesPullingTime="1.814947291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:08.03221823 +0000 UTC m=+48.931014169" watchObservedRunningTime="2026-10-30 01:29:08.032597061 +0000 UTC m=+48.931392993" Oct 30 01:29:08.082025 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.081976 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zdlwv" podStartSLOduration=2.004514031 podStartE2EDuration="4.081961559s" totalImagesPullingTime="2.077447528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:29:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:08.08193242 +0000 UTC m=+48.980728359" watchObservedRunningTime="2026-10-30 01:29:08.081961559 +0000 UTC m=+48.980757497" Oct 30 01:29:08.159379 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.159226 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf"] Oct 30 01:29:08.161599 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:08.161563 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07d2c61_a1ab_4ece_8205_6515c8b8d880.slice/crio-c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0 WatchSource:0}: Error finding container c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0: Status 404 returned error can't find the container with id c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0 Oct 30 01:29:08.948021 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.947984 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:08.950573 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.950504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.968666 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968635 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 01:29:08.968666 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968656 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 01:29:08.968827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968722 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 01:29:08.968827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968751 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 01:29:08.968827 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968794 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bfmqc\"" Oct 30 01:29:08.968960 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968828 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 01:29:08.968960 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968888 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 01:29:08.968960 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968894 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 01:29:08.968960 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968906 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 01:29:08.969059 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.968972 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 01:29:08.975299 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975485 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975485 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975485 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975485 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975781 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975834 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975901 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975901 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.975990 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.975975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:08.986370 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.986345 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:08.994412 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.994383 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gm2cs" event={"ID":"9d69ca43-18c0-4010-87a9-7e3473dfe936","Type":"ContainerStarted","Data":"62fe196ab45825cd4c3740e6ba1cb5ea5cf57634131bdb351e2b2946f36f9c73"} Oct 30 01:29:08.996302 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.996268 2577 generic.go:356] "Generic (PLEG): container finished" podID="76fcebb0-d866-476d-859e-f8b8626a4a79" containerID="1f581c89e6e120b58134876113cf0d27aed7b14f9287297f2ec2d42ce37289bd" exitCode=0 Oct 30 01:29:08.996422 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.996320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mp4k" event={"ID":"76fcebb0-d866-476d-859e-f8b8626a4a79","Type":"ContainerDied","Data":"1f581c89e6e120b58134876113cf0d27aed7b14f9287297f2ec2d42ce37289bd"} Oct 30 01:29:08.997903 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:08.997868 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" event={"ID":"e07d2c61-a1ab-4ece-8205-6515c8b8d880","Type":"ContainerStarted","Data":"c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0"} Oct 30 01:29:09.076449 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076449 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076411 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076656 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076656 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076656 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076656 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076656 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076896 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076896 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076896 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.076896 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.077082 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.077082 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.076947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.077867 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.077440 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.077867 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.077546 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.078895 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.078864 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.080258 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.080232 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.081363 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.081337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.081755 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.081733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.081927 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.081911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.082029 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.082007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.082084 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.082067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.083088 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.083065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.083200 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.083104 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.084943 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.084926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.103270 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.103242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.110653 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.110605 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gm2cs" podStartSLOduration=2.432473601 podStartE2EDuration="5.110590446s" totalImagesPullingTime="2.678116845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:09.10967479 +0000 UTC m=+50.008470731" watchObservedRunningTime="2026-10-30 01:29:09.110590446 +0000 UTC m=+50.009386384" Oct 30 01:29:09.270500 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.270461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:09.598518 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:09.597466 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:09.602580 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:09.602548 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e81463a_be01_4475_94b8_2a84429273b9.slice/crio-e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85 WatchSource:0}: Error finding container e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85: Status 404 returned error can't find the container with id e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85 Oct 30 01:29:10.006706 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.006610 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" event={"ID":"e07d2c61-a1ab-4ece-8205-6515c8b8d880","Type":"ContainerStarted","Data":"3c757ed8c86e09960d9e8ba83843e3bdd042b3998e9be2abd93cf1e53467f485"} Oct 30 01:29:10.006706 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.006703 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" event={"ID":"e07d2c61-a1ab-4ece-8205-6515c8b8d880","Type":"ContainerStarted","Data":"f01e5b05a80d149467bf4453b333da2d659ca398b06081f277df54962e759b62"} Oct 30 01:29:10.007173 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.006736 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" event={"ID":"e07d2c61-a1ab-4ece-8205-6515c8b8d880","Type":"ContainerStarted","Data":"4d7c20e1a5c14eb9fd3ce7282633e5576dca8568c7f3bf585d9c61b1bf2cbff5"} Oct 30 01:29:10.008686 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.008658 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mp4k" event={"ID":"76fcebb0-d866-476d-859e-f8b8626a4a79","Type":"ContainerStarted","Data":"99747181c14d2483c2142ec1e9e92dc3952c506211a02696c9096a948e220eca"} Oct 30 01:29:10.008796 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.008698 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mp4k" event={"ID":"76fcebb0-d866-476d-859e-f8b8626a4a79","Type":"ContainerStarted","Data":"0ef92bd7e889f71aa622c93a58ae055c77b315090ee2035df5777712dbe80e76"} Oct 30 01:29:10.009784 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.009762 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85"} Oct 30 01:29:10.057878 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.057828 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" podStartSLOduration=1.8500722889999999 podStartE2EDuration="3.057815796s" totalImagesPullingTime="1.207743507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:10.056803121 +0000 UTC m=+50.955599059" watchObservedRunningTime="2026-10-30 01:29:10.057815796 +0000 UTC m=+50.956611774" Oct 30 01:29:10.106601 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:10.106543 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8mp4k" podStartSLOduration=2.190224686 podStartE2EDuration="3.106528099s" totalImagesPullingTime="842.700597ms" totalInitContainerRuntime="73.602816ms" isStatefulPod=true podCreationTimestamp="2026-10-30 01:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:10.105634073 +0000 UTC m=+51.004430011" watchObservedRunningTime="2026-10-30 01:29:10.106528099 +0000 UTC m=+51.005324036" Oct 30 01:29:11.014260 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:11.014225 2577 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" exitCode=0 Oct 30 01:29:11.014733 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:11.014313 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} Oct 30 01:29:12.308136 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.308094 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv"] Oct 30 01:29:12.310812 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.310793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.327754 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.327730 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 01:29:12.342098 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.342073 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kg7c7\"" Oct 30 01:29:12.343196 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.343176 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv"] Oct 30 01:29:12.403798 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.403768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.403987 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.403916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24sk\" (UniqueName: \"kubernetes.io/projected/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-kube-api-access-w24sk\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.504275 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.504255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.504393 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.504311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24sk\" (UniqueName: \"kubernetes.io/projected/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-kube-api-access-w24sk\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.506300 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.506278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.529572 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.529548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24sk\" (UniqueName: \"kubernetes.io/projected/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-kube-api-access-w24sk\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.621676 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.621652 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:12.924908 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:12.924881 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv"] Oct 30 01:29:12.926669 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:12.926644 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19919dff_4a1d_470c_9d64_bedcb6bb5a2a.slice/crio-b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5 WatchSource:0}: Error finding container b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5: Status 404 returned error can't find the container with id b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5 Oct 30 01:29:13.022441 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.022407 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" event={"ID":"19919dff-4a1d-470c-9d64-bedcb6bb5a2a","Type":"ContainerStarted","Data":"b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5"} Oct 30 01:29:13.025303 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.025275 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} Oct 30 01:29:13.025431 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.025344 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} Oct 30 01:29:13.025431 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.025382 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} Oct 30 01:29:13.025431 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.025419 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} Oct 30 01:29:13.025553 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.025455 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} Oct 30 01:29:13.934269 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:13.934244 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:14.033842 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:14.033809 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerStarted","Data":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} Oct 30 01:29:14.095945 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:14.095891 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.912794976 podStartE2EDuration="6.095874448s" totalImagesPullingTime="3.169517836s" totalInitContainerRuntime="13.561636ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:14.095003425 +0000 UTC m=+54.993799386" watchObservedRunningTime="2026-10-30 01:29:14.095874448 +0000 UTC m=+54.994670385" Oct 30 01:29:15.041961 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:15.041922 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" event={"ID":"19919dff-4a1d-470c-9d64-bedcb6bb5a2a","Type":"ContainerStarted","Data":"f86ab23435568a74091d48199c2647cef8f1fc7436edabcf7ebfa3b1693fc372"} Oct 30 01:29:15.042516 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:15.042418 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:15.047115 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:15.047096 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:15.090553 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:15.090505 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" podStartSLOduration=1.861795029 podStartE2EDuration="3.09049259s" totalImagesPullingTime="1.228697561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:15.089347253 +0000 UTC m=+55.988143190" watchObservedRunningTime="2026-10-30 01:29:15.09049259 +0000 UTC m=+55.989288528" Oct 30 01:29:18.000911 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:18.000877 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:21.956357 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:21.956328 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nc6vm"] Oct 30 01:29:21.959766 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:21.959740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:21.977726 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:21.977701 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qp94x\"" Oct 30 01:29:21.993299 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:21.993275 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nc6vm"] Oct 30 01:29:22.021312 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.021284 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 01:29:22.021426 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.021334 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 01:29:22.085199 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.085178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec32c8f1-eb5a-4a91-af80-f41d281d672e-tmp\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.085335 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.085229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smb44\" (UniqueName: \"kubernetes.io/projected/ec32c8f1-eb5a-4a91-af80-f41d281d672e-kube-api-access-smb44\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.186002 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.185969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smb44\" (UniqueName: \"kubernetes.io/projected/ec32c8f1-eb5a-4a91-af80-f41d281d672e-kube-api-access-smb44\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.186173 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.186094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec32c8f1-eb5a-4a91-af80-f41d281d672e-tmp\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.186526 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.186505 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec32c8f1-eb5a-4a91-af80-f41d281d672e-tmp\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.212399 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.212351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smb44\" (UniqueName: \"kubernetes.io/projected/ec32c8f1-eb5a-4a91-af80-f41d281d672e-kube-api-access-smb44\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.269579 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.269547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:22.590978 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:22.590895 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nc6vm"] Oct 30 01:29:22.593050 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:22.593024 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec32c8f1_eb5a_4a91_af80_f41d281d672e.slice/crio-c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe WatchSource:0}: Error finding container c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe: Status 404 returned error can't find the container with id c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe Oct 30 01:29:23.079030 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:23.078989 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nc6vm" event={"ID":"ec32c8f1-eb5a-4a91-af80-f41d281d672e","Type":"ContainerStarted","Data":"c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe"} Oct 30 01:29:24.406267 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.406222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:24.406267 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.406271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:24.422625 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.422594 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:29:24.429612 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.429583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:24.437275 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.437253 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:29:24.449800 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.449777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:24.507342 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.507305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:24.523954 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.523925 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:29:24.544543 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.544515 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:29:24.551829 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.551792 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:24.661165 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.661075 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:29:24.665274 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.665240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:24.668802 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.668781 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:29:24.673463 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.673445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:24.677011 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.676990 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:29:24.681100 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.681081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:24.981285 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.981260 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jt6kw"] Oct 30 01:29:24.983362 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:24.983329 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141993bb_335c_4e9c_b08e_da41dd9d7f0f.slice/crio-c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9 WatchSource:0}: Error finding container c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9: Status 404 returned error can't find the container with id c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9 Oct 30 01:29:24.992259 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:24.992237 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9qlzc"] Oct 30 01:29:24.995367 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:24.995345 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131e7807_d6f8_4104_836e_33c6dcbbc286.slice/crio-8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da WatchSource:0}: Error finding container 8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da: Status 404 returned error can't find the container with id 8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da Oct 30 01:29:25.002663 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:25.002640 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xv85r"] Oct 30 01:29:25.004709 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:25.004689 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4999ae_bde0_4c8a_9652_4ea7811afdaa.slice/crio-6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381 WatchSource:0}: Error finding container 6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381: Status 404 returned error can't find the container with id 6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381 Oct 30 01:29:25.087718 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:25.087678 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jt6kw" event={"ID":"141993bb-335c-4e9c-b08e-da41dd9d7f0f","Type":"ContainerStarted","Data":"c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9"} Oct 30 01:29:25.089291 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:25.089048 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xv85r" event={"ID":"ba4999ae-bde0-4c8a-9652-4ea7811afdaa","Type":"ContainerStarted","Data":"6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381"} Oct 30 01:29:25.090240 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:25.090218 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9qlzc" event={"ID":"131e7807-d6f8-4104-836e-33c6dcbbc286","Type":"ContainerStarted","Data":"8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da"} Oct 30 01:29:26.833298 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.833251 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-545767f87d-ccnvd"] Oct 30 01:29:26.837608 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.836555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.854872 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.854844 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 01:29:26.854872 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.854862 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 01:29:26.855038 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.855012 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 01:29:26.871250 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.869683 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 01:29:26.871250 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.869745 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fskhp\"" Oct 30 01:29:26.871250 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.869924 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 01:29:26.871250 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.871200 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545767f87d-ccnvd"] Oct 30 01:29:26.925212 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.925357 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.925357 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.925357 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.925357 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.925512 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:26.925512 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:26.925402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026488 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026651 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.026845 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.026662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.027457 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.027413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.027787 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.027741 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.027902 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.027786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.027902 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.027895 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.035784 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.035756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.043855 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.043830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.057746 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.057714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.103918 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.103830 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xv85r" event={"ID":"ba4999ae-bde0-4c8a-9652-4ea7811afdaa","Type":"ContainerStarted","Data":"b8d907dd15e4d412872562b9d259859faa38080711159edc0d5ac4b9bda91a7f"} Oct 30 01:29:27.103918 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.103906 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xv85r" event={"ID":"ba4999ae-bde0-4c8a-9652-4ea7811afdaa","Type":"ContainerStarted","Data":"51e3834fbc364c891c68c75aa322b6226289a987e31e004d33cdd48325c3c04c"} Oct 30 01:29:27.152052 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.152019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:27.483201 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.483084 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xv85r" podStartSLOduration=66.370109385 podStartE2EDuration="1m7.483063657s" totalImagesPullingTime="1.112954272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:27.152396708 +0000 UTC m=+68.051192646" watchObservedRunningTime="2026-10-30 01:29:27.483063657 +0000 UTC m=+68.381859595" Oct 30 01:29:27.483435 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:27.483417 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545767f87d-ccnvd"] Oct 30 01:29:27.487430 ip-10-0-143-27 kubenswrapper[2577]: W1030 01:29:27.487404 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b65ce43_3ae8_461b_b703_fbc0b37645ea.slice/crio-ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c WatchSource:0}: Error finding container ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c: Status 404 returned error can't find the container with id ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c Oct 30 01:29:28.111221 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:28.111181 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545767f87d-ccnvd" event={"ID":"9b65ce43-3ae8-461b-b703-fbc0b37645ea","Type":"ContainerStarted","Data":"ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c"} Oct 30 01:29:29.117873 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:29.117831 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9qlzc" event={"ID":"131e7807-d6f8-4104-836e-33c6dcbbc286","Type":"ContainerStarted","Data":"216423c6ac387fee380e8447d45ba359d3b87c4fb715f5326e748f017ee511fb"} Oct 30 01:29:29.118501 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:29.117999 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:29.164028 ip-10-0-143-27 kubenswrapper[2577]: I1030 01:29:29.163968 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9qlzc" podStartSLOduration=65.188834663 podStartE2EDuration="1m9.163950224s" totalImagesPullingTime="3.975115561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:29.163659874 +0000 UTC m=+70.062455815" watchObservedRunningTime="2026-10-30 01:29:29.163950224 +0000 UTC m=+70.062746163" Oct 30 01:29:31.529679 ip-10-0-143-27 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 01:29:31.545614 ip-10-0-143-27 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 01:29:31.545801 ip-10-0-143-27 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 01:29:31.545863 ip-10-0-143-27 systemd[1]: kubelet.service: Consumed 3.928s CPU time. Oct 30 01:29:31.574076 ip-10-0-143-27 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:29:31.637043 ip-10-0-143-27 kubenswrapper[6275]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:29:31.637043 ip-10-0-143-27 kubenswrapper[6275]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:29:31.637043 ip-10-0-143-27 kubenswrapper[6275]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:29:31.637043 ip-10-0-143-27 kubenswrapper[6275]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:29:31.639447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639356 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.639447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639443 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.639447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639448 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.639447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639453 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639458 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639462 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639466 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639470 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639475 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639479 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639483 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639486 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639490 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639495 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639499 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639503 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639507 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639512 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639515 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639519 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639524 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639527 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.639726 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639531 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639535 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639539 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639543 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639547 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639551 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639556 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639560 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639564 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639567 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639572 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639576 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639580 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639584 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639589 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639594 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639599 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639612 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639617 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639621 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.640587 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639626 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639630 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639634 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639640 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639647 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639652 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639657 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639662 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639667 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639671 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639676 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639680 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639684 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639689 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639694 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639698 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639703 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639708 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639712 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.641493 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639717 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639721 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639725 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639729 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639733 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639737 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639742 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639746 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639752 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639757 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639761 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639766 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639771 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639775 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639780 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639784 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639788 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639792 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639796 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639801 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.642362 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639805 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639810 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639814 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639819 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639824 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639828 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639832 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639837 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639841 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639847 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639851 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639855 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639860 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639873 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639878 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639882 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639886 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639893 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639897 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639902 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.643291 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639906 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639910 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639914 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639919 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.639924 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641223 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641234 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641239 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641244 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641249 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641253 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641258 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641262 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641266 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641270 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641276 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641281 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641285 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641290 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641294 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.644217 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641298 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641303 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641308 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641312 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641317 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641321 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641325 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641330 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641334 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641338 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641342 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641347 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641351 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641355 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641360 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641364 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641368 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641376 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641383 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.645143 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641389 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641393 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641398 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641404 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641408 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641414 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641420 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641425 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641430 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641434 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641438 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641443 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641447 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641451 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641456 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641461 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641466 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641471 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641475 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641480 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.645936 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641484 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641488 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641492 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641496 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641500 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641504 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641508 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641512 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641517 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641522 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641527 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641533 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641537 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641542 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641546 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641550 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641555 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641559 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641564 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641568 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.646546 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641572 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641577 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641582 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641586 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641590 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641595 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641600 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641604 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641609 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641613 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641617 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641622 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641626 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641631 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641635 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641640 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641644 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641648 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641652 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641657 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.647071 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641661 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641665 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641669 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641673 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641678 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641682 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641686 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641696 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641701 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641705 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641709 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.641713 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642584 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642596 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642603 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642609 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642614 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642618 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642623 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642628 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.647609 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642632 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642636 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642641 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642645 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642650 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642654 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642659 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642663 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642667 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642672 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642676 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642680 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642685 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642689 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642693 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642698 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642703 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642707 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642711 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.648250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642718 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642722 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642727 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642731 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642735 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642739 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642744 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642748 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642752 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642756 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642760 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642765 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642769 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642773 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642778 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642782 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642787 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642791 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642796 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.649086 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642800 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642804 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642808 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642812 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642817 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642821 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642825 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642829 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642834 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642838 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642842 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642847 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642852 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642858 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642862 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642866 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642870 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642876 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642881 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.649626 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642885 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642889 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642894 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642898 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642903 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642907 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642911 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642915 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642919 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642923 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642928 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642932 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642936 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642940 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642944 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642949 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642953 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642957 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642961 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642965 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.650100 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642970 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642974 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642981 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642988 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642993 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.642999 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643005 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643010 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643015 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643019 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643024 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643028 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643033 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643037 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643041 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643045 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643049 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643054 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643058 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.650655 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643063 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.643069 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643211 6275 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643223 6275 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643233 6275 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643240 6275 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643247 6275 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643253 6275 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643260 6275 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643266 6275 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643272 6275 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643276 6275 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643282 6275 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643287 6275 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643292 6275 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643297 6275 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643301 6275 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643306 6275 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643310 6275 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643319 6275 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643327 6275 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643332 6275 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643337 6275 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643342 6275 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:29:31.651146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643348 6275 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643354 6275 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643359 6275 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643364 6275 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643369 6275 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643374 6275 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643379 6275 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643384 6275 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643389 6275 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643394 6275 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643401 6275 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643406 6275 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643411 6275 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643416 6275 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643421 6275 flags.go:64] FLAG: --enable-server="true" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643426 6275 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643433 6275 flags.go:64] FLAG: --event-burst="100" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643438 6275 flags.go:64] FLAG: --event-qps="50" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643442 6275 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643447 6275 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643452 6275 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643459 6275 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643464 6275 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643469 6275 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643474 6275 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:29:31.651750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643479 6275 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643484 6275 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643496 6275 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643501 6275 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643506 6275 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643512 6275 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643516 6275 flags.go:64] FLAG: --feature-gates="" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643523 6275 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643528 6275 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643533 6275 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643539 6275 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643545 6275 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643551 6275 flags.go:64] FLAG: --help="false" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643555 6275 flags.go:64] FLAG: --hostname-override="ip-10-0-143-27.ec2.internal" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643561 6275 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643566 6275 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643571 6275 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643577 6275 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643582 6275 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643587 6275 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643592 6275 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643597 6275 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643602 6275 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:29:31.652395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643607 6275 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643612 6275 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643617 6275 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643621 6275 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643626 6275 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643631 6275 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643636 6275 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643641 6275 flags.go:64] FLAG: --lock-file="" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643645 6275 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643650 6275 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643655 6275 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643666 6275 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643671 6275 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643677 6275 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643683 6275 flags.go:64] FLAG: --logging-format="text" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643688 6275 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643693 6275 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643698 6275 flags.go:64] FLAG: --manifest-url="" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643703 6275 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643709 6275 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643714 6275 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643721 6275 flags.go:64] FLAG: --max-pods="110" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643726 6275 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643732 6275 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643737 6275 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:29:31.652970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643742 6275 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643746 6275 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643751 6275 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643756 6275 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643767 6275 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643772 6275 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643777 6275 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643782 6275 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643787 6275 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643792 6275 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643796 6275 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643801 6275 flags.go:64] FLAG: --port="10250" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643806 6275 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643811 6275 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e80abfd091b34d36" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643816 6275 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643821 6275 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643826 6275 flags.go:64] FLAG: --register-node="true" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643832 6275 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643842 6275 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643847 6275 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643852 6275 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643857 6275 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643872 6275 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643878 6275 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643883 6275 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643888 6275 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:29:31.653600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643892 6275 flags.go:64] FLAG: --runonce="false" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643897 6275 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643902 6275 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643907 6275 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643912 6275 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643917 6275 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643922 6275 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643927 6275 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643932 6275 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643936 6275 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643941 6275 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643945 6275 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643950 6275 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643955 6275 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643960 6275 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643965 6275 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643974 6275 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643979 6275 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643983 6275 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643990 6275 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.643995 6275 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644000 6275 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644005 6275 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644010 6275 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644017 6275 flags.go:64] FLAG: --v="2" Oct 30 01:29:31.654269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644025 6275 flags.go:64] FLAG: --version="false" Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644031 6275 flags.go:64] FLAG: --vmodule="" Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644037 6275 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644044 6275 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644275 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644283 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644288 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644292 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644297 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644302 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644306 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644311 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644316 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644320 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644324 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644328 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644333 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644337 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644342 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644346 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.654923 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644350 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644355 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644359 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644364 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644368 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644372 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644377 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644381 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644386 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644390 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644394 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644401 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644406 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644413 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644419 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644424 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644430 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644435 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644440 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.655585 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644444 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644448 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644452 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644457 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644461 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644465 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644470 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644474 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644479 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644483 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644488 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644492 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644496 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644501 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644505 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644510 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644514 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644518 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644522 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644527 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.656082 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644531 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644536 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644540 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644544 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644554 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644559 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644564 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644568 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644573 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644577 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644582 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644587 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644591 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644595 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644600 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644604 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644608 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644613 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644617 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.656590 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644621 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644625 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644630 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644634 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644638 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644642 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644647 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644651 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644656 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644660 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644665 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644669 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644674 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644678 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644682 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644687 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644691 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644698 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644702 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644707 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.657447 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644711 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644716 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644720 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644727 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644734 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644740 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644745 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644750 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644754 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644759 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644763 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644767 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.644809 6275 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644865 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644872 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.658310 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644877 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644882 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644887 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644891 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644896 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644900 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644904 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644909 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644916 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644922 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644928 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644932 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644937 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644941 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644947 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644951 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644956 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644961 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644966 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.658770 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644970 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644974 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644978 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644984 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644988 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644992 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.644996 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645001 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645005 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645010 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645014 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645021 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645025 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645029 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645033 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645038 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645042 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645046 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645051 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645055 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.659349 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645060 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645065 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645069 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645073 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645077 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645081 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645086 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645092 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645098 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645103 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645107 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645111 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645116 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645135 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645139 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645144 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645149 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645153 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645157 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.659914 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645161 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645166 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645170 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645174 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645181 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645186 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645191 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645195 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645199 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645204 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645208 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645213 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645217 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645221 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645227 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645231 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645236 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645240 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645247 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645254 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.660408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645261 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645266 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645270 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645275 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645279 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645283 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645288 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645292 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645296 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645300 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645306 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645310 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645315 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645319 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645325 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645329 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645335 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645340 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645344 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645348 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.660891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645353 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645357 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645361 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645366 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645370 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.645374 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.650580 6275 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.650598 6275 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.650615 6275 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.650620 6275 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650677 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650682 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650687 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650691 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650694 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650697 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650701 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650704 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.661386 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650706 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650709 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650712 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650715 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650718 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650721 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650723 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650731 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650734 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650737 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650739 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650742 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650744 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650747 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650750 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650753 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650756 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650759 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650762 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650764 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.661843 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650767 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650770 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650773 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650775 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650778 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650781 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650783 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650786 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650788 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650791 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650794 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650797 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650799 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650802 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650804 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650806 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650809 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650811 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650815 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650817 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.662406 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650820 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650823 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650825 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650828 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650831 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650834 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650837 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650839 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650842 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650845 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650848 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650850 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650853 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650855 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650859 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650863 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650866 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650869 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650872 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.662877 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650875 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650878 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650880 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650883 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650886 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650888 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650891 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650894 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650896 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650899 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650901 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650904 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650907 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650910 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650912 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650915 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650919 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650923 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650928 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650932 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.663408 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650937 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650939 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650942 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650945 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650948 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650950 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650953 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650955 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650959 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650961 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650964 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650966 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650969 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650972 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650974 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650977 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650979 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650982 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.663896 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.650984 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.651012 6275 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651192 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651198 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651202 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651206 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651209 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651212 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651215 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651218 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651221 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651224 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651226 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651233 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651235 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651238 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.664341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651241 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651244 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651247 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651250 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651252 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651255 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651257 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651260 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651263 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651265 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651268 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651271 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651273 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651276 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651279 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651281 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651284 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651286 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.664752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651289 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651291 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651294 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651297 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651299 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651302 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651306 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651309 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651312 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651315 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651318 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651322 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651325 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651328 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651330 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651333 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651336 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651339 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651342 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651345 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.665192 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651347 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651350 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651353 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651357 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651361 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651364 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651367 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651369 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651372 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651375 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651377 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651380 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651382 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651385 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651387 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651390 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651392 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651395 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651398 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651401 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.665722 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651404 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651406 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651409 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651413 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651415 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651418 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651421 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651423 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651426 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651429 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651431 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651434 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651436 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651439 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651441 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651444 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651447 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651449 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651452 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.666752 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651455 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651457 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651460 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651462 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651465 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651469 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651471 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651474 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651477 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651479 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651482 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651484 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651487 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651490 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651494 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.651518 6275 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:29:31.667539 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651550 6275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651556 6275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651560 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651563 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651566 6275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651569 6275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651571 6275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651574 6275 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651576 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651579 6275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651582 6275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651586 6275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651590 6275 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651593 6275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651596 6275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651599 6275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651602 6275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651605 6275 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651608 6275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:31.667947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651610 6275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651613 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651617 6275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651620 6275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651622 6275 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651625 6275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651628 6275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651630 6275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651633 6275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651636 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651638 6275 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651642 6275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651644 6275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651647 6275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651651 6275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651653 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651656 6275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651659 6275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651662 6275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651664 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:31.668464 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651667 6275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651669 6275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651672 6275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651675 6275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651677 6275 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651680 6275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651683 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651685 6275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651688 6275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651691 6275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651693 6275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651696 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651698 6275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651701 6275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651705 6275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651708 6275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651711 6275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651713 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651716 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651718 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:31.668947 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651721 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651723 6275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651727 6275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651731 6275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651737 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651740 6275 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651744 6275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651746 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651749 6275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651752 6275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651754 6275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651757 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651760 6275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651763 6275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651766 6275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651768 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651771 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651774 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651777 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:31.669458 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651779 6275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651782 6275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651784 6275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651787 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651789 6275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651792 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651794 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651798 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651801 6275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651804 6275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651806 6275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651809 6275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651812 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651814 6275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651817 6275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651819 6275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651822 6275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651824 6275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:31.669919 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651827 6275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651831 6275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651834 6275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651837 6275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651840 6275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651843 6275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651847 6275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651850 6275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651853 6275 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:31.651855 6275 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.651921 6275 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.652916 6275 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.652977 6275 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.653185 6275 server.go:1014] "Starting client certificate rotation" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.653321 6275 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.653377 6275 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:23:20 +0000 UTC" deadline="2028-06-24 15:26:38.431338645 +0000 UTC" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.653405 6275 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14485h57m6.777936004s" Oct 30 01:29:31.670502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.653676 6275 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:29:31.670923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.654814 6275 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:29:31.670923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.656067 6275 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:29:31.670923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.658393 6275 log.go:25] "Validated CRI v1 image API" Oct 30 01:29:31.670923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.659160 6275 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:29:31.670923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.661283 6275 fs.go:135] Filesystem UUIDs: map[0f2a8570-a2fc-4e18-a79a-5f4b5c41f552:/dev/nvme0n1p4 2151f9e5-99a8-4728-8619-ed78212dacfb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 01:29:31.671091 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.661301 6275 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11/userdata/shm major:0 minor:53 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85/userdata/shm major:0 minor:280 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9/volumes/kubernetes.io~projected/kube-api-access-7p5kq:{mountpoint:/var/lib/kubelet/pods/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9/volumes/kubernetes.io~projected/kube-api-access-7p5kq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volumes/kubernetes.io~projected/kube-api-access-h7pqj:{mountpoint:/var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volumes/kubernetes.io~projected/kube-api-access-h7pqj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/131e7807-d6f8-4104-836e-33c6dcbbc286/volumes/kubernetes.io~projected/kube-api-access-sgf5g:{mountpoint:/var/lib/kubelet/pods/131e7807-d6f8-4104-836e-33c6dcbbc286/volumes/kubernetes.io~projected/kube-api-access-sgf5g major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141993bb-335c-4e9c-b08e-da41dd9d7f0f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/141993bb-335c-4e9c-b08e-da41dd9d7f0f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19919dff-4a1d-470c-9d64-bedcb6bb5a2a/volumes/kubernetes.io~projected/kube-api-access-w24sk:{mountpoint:/var/lib/kubelet/pods/19919dff-4a1d-470c-9d64-bedcb6bb5a2a/volumes/kubernetes.io~projected/kube-api-access-w24sk major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19919dff-4a1d-470c-9d64-bedcb6bb5a2a/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/19919dff-4a1d-470c-9d64-bedcb6bb5a2a/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/467074bb-b3a4-444d-a8b9-9aae4f9d9d56/volumes/kubernetes.io~projected/kube-api-access-g7pmk:{mountpoint:/var/lib/kubelet/pods/467074bb-b3a4-444d-a8b9-9aae4f9d9d56/volumes/kubernetes.io~projected/kube-api-access-g7pmk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8/volumes/kubernetes.io~projected/kube-api-access-jg9x6:{mountpoint:/var/lib/kubelet/pods/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8/volumes/kubernetes.io~projected/kube-api-access-jg9x6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/alertmanager/9 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~projected/kube-api-access-nscr6:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~projected/kube-api-access-nscr6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~projected/tls-assets major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/config-volume major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/web-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~projected/kube-api-access-2mjk5:{mountpoint:/var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~projected/kube-api-access-2mjk5 major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~empty-dir/tmp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~projected/kube-api-access-j2c22:{mountpoint:/var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~projected/kube-api-access-j2c22 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f2c416d-69b2-49ef-a2e3-230c5935d67b/volumes/kubernetes.io~projected/kube-api-access-cbb8q:{mountpoint:/var/lib/kubelet/pods/7f2c416d-69b2-49ef-a2e3-230c5935d67b/volumes/kubernetes.io~projected/kube-api-access-cbb8q major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f2c416d-69b2-49ef-a2e3-230c5935d67b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7f2c416d-69b2-49ef-a2e3-230c5935d67b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9712dd02-a8c8-402f-bfcd-8e149dc4835b/volumes/kubernetes.io~projected/kube-api-access-zxbnk:{mountpoint:/var/lib/kubelet/pods/9712dd02-a8c8-402f-bfcd-8e149dc4835b/volumes/kubernetes.io~projected/kube-api-access-zxbnk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~projected/kube-api-access-dkx87:{mountpoint:/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~projected/kube-api-access-dkx87 major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c859dda-f9c4-4c70-b6d2-e72b990dc465/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9c859dda-f9c4-4c70-b6d2-e72b990dc465/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d69ca43-18c0-4010-87a9-7e3473dfe936/volumes/kubernetes.io~projected/kube-api-access-8sbr9:{mountpoint:/var/lib/kubelet/pods/9d69ca43-18c0-4010-87a9-7e3473dfe936/volumes/kubernetes.io~projected/kube-api-access-8sbr9 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d69ca43-18c0-4010-87a9-7e3473dfe936/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9d69ca43-18c0-4010-87a9-7e3473dfe936/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/volumes/kubernetes.io~projected/kube-api-access-n8gmk:{mountpoint:/var/lib/kubelet/pods/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/volumes/kubernetes.io~projected/kube-api-access-n8gmk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba4999ae-bde0-4c8a-9652-4ea7811afdaa/volumes/kubernetes.io~projected/kube-api-access-pt882:{mountpoint:/var/lib/kubelet/pods/ba4999ae-bde0-4c8a-9652-4ea7811afdaa/volumes/kubernetes.io~projected/kube-api-access-pt882 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba4999ae-bde0-4c8a-9652-4ea7811afdaa/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ba4999ae-bde0-4c8a-9652-4ea7811afdaa/volumes/kubernetes.io~secret/metrics-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cded0e24-b028-447a-9bad-beda2dd20dbb/volumes/kubernetes.io~projected/kube-api-access-zgv67:{mountpoint:/var/lib/kubelet/pods/cded0e24-b028-447a-9bad-beda2dd20dbb/volumes/kubernetes.io~projected/kube-api-access-zgv67 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d14d41c9-2491-4d02-83aa-356a6e8aa1e5/volumes/kubernetes.io~projected/kube-api-access-nlrnd:{mountpoint:/var/lib/kubelet/pods/d14d41c9-2491-4d02-83aa-356a6e8aa1e5/volumes/kubernetes.io~projected/kube-api-access-nlrnd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~projected/kube-api-access-7hf5v:{mountpoint:/var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~projected/kube-api-access-7hf5v major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec32c8f1-eb5a-4a91-af80-f41d281d672e/volumes/kubernetes.io~projected/kube-api-access-smb44:{mountpoint:/var/lib/kubelet/pods/ec32c8f1-eb5a-4a91-af80-f41d281d672e/volumes/kubernetes.io~projected/kube-api-access-smb44 major:0 minor:343 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4fd8d62e64134463c0ce2debd72e4b87ffa882e602451fb73b58558a14385869/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1c63e7290e46a4e486a4e4c85179b8948a2a8a9e1ae0249a5ca9e39a216c54fc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5c787327eb6a943e5a974ce9e7a31a7a2c308dfdb1905736a015daa381a86521/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/81e2fa36a66a8138c71df097bc05611a10de3569912205bfff46f91527f031f7/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e4811f35dbb0bed51272174ec186352cb737ccde6326b844273a322ffad25207/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/76b22b58648f5a863ad15f9dda9f87878758fd24c295d2027365bc847096ca3d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/edebfa75f0c0b97e165466b0aa30f43eae5bdc12011a7fa4cc3804bd48bf043f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b44f1c558090b22358ffd873f5691f390cb07b36c9ebfd6f9f4caa9bed8ec6f3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/d0fed60aa80e334d9945873f10d492e59fc77215c21ee0ad82161fe601a13679/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1f9a464da64b21c20d0fe01d69a3ea28a54ff512522f0c2d284a81418c6293e7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/132a7806f82c3f635feadd32a1d6925ad0f9e5d046ddbb1de38c27f94397ea79/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/06b81dc8871206ab5dd3e2334fe0e07e9678c31caa5cd1427e5bc6cfbebc42ac/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b6bdc22ffc193b077f691f52103bc5097e0aaea80e1b879a5af02786082c08cf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2c7d84f57dd8fea4cbaf997e940bb34420b82ea7bb5fb325afe221831c0b0bd5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/2e1c9d3ab1cb5edd56e289f1b73920bfea163aee52b4378710f42225bbcc9b48/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7c43b07747f646789ddbb630b237f7daf5e014d4dfb77e3cc0e5983a4cd94e08/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c2151fc41f5e0a6c2a6272863cf74ef2dec4aecb4d2308b27c7e684a79efd24c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ea2870863076869043ed8b4649a2957b267c1ffd55facebb149b929a09d56f4f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/009b60a699dd87671e8a525ec967b035d4da8dceeb64ce83592ee44c2dd28d29/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/340c0f6da0dfc756ed9b4a9acb9521a9dd74feee964429561640346739c0a54f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/180ac2cbf1bdf09461e7562f0baf45c2e65ff6c7a79d65852aa2e5306f8f8de5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/72868d38b8fc916aa54435a3561e2f2d7ea069da89a2954c316d5266afaf371b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ec6e1edd770fd16205e277967eb7aad6d6fc01ac2196ba7e1766c7a338e0dd9a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e6bc57c50b422f08b62b06060095fc029963717013ea3ba805a1c1a94ac2f9db/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3e4995253459f1232292178f6458d67eef9d3f1a7db412f6c31971e79370b538/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/552221fb80c82724e28a52d484ab57fa07e7ed6324cda13f40422e705d66476e/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/65dc3fd1935486f3b42b4c3260fb2d4eb08fd05ec761ada6c5891edf7957acfb/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b8a45b73dd6c8c22798c3f22da9a59305654efad6cc7362138847d97aa5f3597/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/6bc72e45421b495a6c2a74833e0fa6627e6a53ba58f4954263430e45a3e7bb46/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/9f15b02248c301061b51deb7f7702c03a671c0a6a2027d5d810754abd4577a42/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/ff17791126df9712c428b433a6dbb279c79841f6049791989b1c881cbcb0d1a1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/689c50e6c978c2b80c30d14bbbb4571270e40510424f9f6ef08f319e2c696d41/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c6fe4710845485317e86b01cbad90feac0fd11ecbf04ad4403001cae40a23886/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/42191049494d00269bfff07f1339d7c4a4041709238583fc1aac28f1b1fbe5ab/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/1a000c80c82b51f1443ff0e8863bbda51aa6a02e57c391c3db235ae1f5b47dd4/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/c0b843727e82c871a77101974c6dc9dfb0a2b96e163d36916e45355c1b7fcbaa/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/97b59918d0754805b41c822ef663900fc97f5202586ec087c16b2323a9b55ccb/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/962e6c633d9b0381ff91050a76ba91df8577dc2fb76191356f914111316cd7d4/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/8e7c65cd2474cb849f94a8c488dc24f94d77ab21642695f861cf714e23f3ec8e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/723b076298e9eecdfc8b046fea813281813b005b707240bf8b44cd7590859d7b/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/42688f930efe7895b99e26ddf414fa9c69162ca02cba42cc689693758a7a5828/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/5090354ac3793b045523c38a94c5dd2390ca8e397fe78e02686f99f2158cddfe/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/4cc2c9105d1730c205a1a8a48bd4607a563d9e3341c06de5ea28c5293046773d/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/e19c97d1a26987142385ee13e093b8f3747b32ba68f04d1ade18aaa6eb209e66/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/94d8a73f2dc27c7108956b91e9bb911c80571143031353f6ff90331579148550/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a5e2b0af577dfc85f0c6c4606f9c791092cb31adb1be8daab772b68c22e1208b/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/c26b6e088f4cae2c2aea08d1ea9c21b49002c8708d261a8e6ac2fc6de416cdf0/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/eb8d759b645fa6c120331efcc21710b0ce4058462da1ecd561e4cc4da4974276/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/cfd01ab07aabda298729880e88e79176099c0823609a6e5586de61d622c2454a/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/9de02463412fd40ec7b900380365a6dc148d6ff186953ea74bffca7a5689774b/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/1b979f50489dfcdbefe40ac557caad13e9e88714253fe16e9394cdfa04275c3c/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/b1da7e830ad037223170043b875d185830c523b33dd37eb2e1ec3aa55217e5dd/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/062777dd1166afe771d9ce5c11deef8be6c9f743c28531bf9933d6ace878ac2c/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/b841a1d3cc4a067ac432447d7ac251eeecfcefdbfb7594e1c52e2b0276dc6e5c/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/76064c48e217f2802f3a78dc4bc776937df164e701cecd626b3434778bcd250b/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/46fb32515a385bba574076eb469ed126adbec235f92aa24dd4d3e525ac57b6a5/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-54:{mountpoint:/var/lib/containers/storage/overlay/a4088dade1e907424d58ba64371b2565f6c37294809e6188f4bfa9c4696f52c2/merged major:0 minor:54 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/de47374cf4083cefd153beff21c1c27f3928cbdbfd6f872ed598ae4e20f6d0f7/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d10d4f4646b4050eb7b0540f6b58d2707f735cf3c22f1e007a41a7046195d23c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f0236816e8bed801c3c3fc3a559fd68776c8998125912a6ce167b12ba4d53ae2/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/cfa88d2677ee8de46600931902f1eb8b500f659c010671de27f6d4cecbc5f06b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9b532e7bf1349ce888a9506078b456bb83beb6465916766f0af704bce6e570fd/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/66f64e8b6ce9c86754630772bd9bcdbdaebf58655a36cec28898743a75603793/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/883d5d8e387eb99929c16080b19c9e2372b0b248da43cd08b140050f563f1915/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c424f8cf9847e90f747f92b3ea54d60b8a62ca0518e6e9d37f821a00fbf16e2c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cd34f4ed981c69799bb2c4354ac3395090dfe4c37ca48be008865a62fa12734c/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670052 6275 manager.go:217] Machine: {Timestamp:2026-10-30 01:29:31.669582833 +0000 UTC m=+0.061390458 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110269 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a5001aabf7e286208b4dcc48af2a3 SystemUUID:ec2a5001-aabf-7e28-6208-b4dcc48af2a3 BootID:55071aca-0de4-4905-a831-755f1c492b6c Filesystems:[{Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~projected/kube-api-access-2mjk5 DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19919dff-4a1d-470c-9d64-bedcb6bb5a2a/volumes/kubernetes.io~projected/kube-api-access-w24sk DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c859dda-f9c4-4c70-b6d2-e72b990dc465/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cded0e24-b028-447a-9bad-beda2dd20dbb/volumes/kubernetes.io~projected/kube-api-access-zgv67 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b70502dd1a1eb9af4d812b21deff12f4a70a8afd0f6bb023de0a09b77a41a016/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/131e7807-d6f8-4104-836e-33c6dcbbc286/volumes/kubernetes.io~projected/kube-api-access-sgf5g DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be06ec4f6c97a84d200d6fdc1b0520305913d1944f27f9d9a3b78a33c8e47f77/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d69ca43-18c0-4010-87a9-7e3473dfe936/volumes/kubernetes.io~projected/kube-api-access-8sbr9 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d14d41c9-2491-4d02-83aa-356a6e8aa1e5/volumes/kubernetes.io~projected/kube-api-access-nlrnd DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~projected/kube-api-access-7hf5v DeviceMajor:0 DeviceMinor:244 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f2c416d-69b2-49ef-a2e3-230c5935d67b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~projected/kube-api-access-j2c22 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/volumes/kubernetes.io~projected/kube-api-access-n8gmk DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/467074bb-b3a4-444d-a8b9-9aae4f9d9d56/volumes/kubernetes.io~projected/kube-api-access-g7pmk DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b73a4cb19ac2b55c9cc16d43f6ad5761e170482aa6c6fdf0422f18cc20c4aa9/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~projected/kube-api-access-dkx87 DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-54 DeviceMajor:0 DeviceMinor:54 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8/volumes/kubernetes.io~projected/kube-api-access-jg9x6 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2011b10f19f1d207b8bb301b20d7fecf84d75ff4e93767147dac7bf2c381c3fd/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b00aef3cb3d02d94f72d93dc2825fcecfe0ce573c7b579b6f73ad6d1365a61b5/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b9c472d469007d98b30f0af5c0097950f02fcf2b16cbf5abfb69e7f06da1381/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76fcebb0-d866-476d-859e-f8b8626a4a79/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba4999ae-bde0-4c8a-9652-4ea7811afdaa/volumes/kubernetes.io~projected/kube-api-access-pt882 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d69ca43-18c0-4010-87a9-7e3473dfe936/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c10bc50b15f6856412d02697cb1ad17716fb55f8000f8089e57f841b615c15e9/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c13b30394881c2a057e75e21ce9a96262a6433898aedd2bab1adac032bd04aa/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e07d2c61-a1ab-4ece-8205-6515c8b8d880/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5f60be588a5974f05249ca47906f4bf2ae7c206ba5056affd2f6adf1dc52e9d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/990a8ac1e764f81984d02b0cf3137a5d780cecc61ac96ec70b3ed51704c92a0c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8b096ecebb97dda38d9cee66a10a336d39ac1968e59e6a5db7a21d89e7677c0/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141993bb-335c-4e9c-b08e-da41dd9d7f0f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:379 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e872c2b43d1efad51cc4a0fc18fe8124c86aa276172c1892770b996bc4f6420e/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4d09783ad02d9d14ff3d16d0278baf972d09d0c706226a03868db7b4add85fe/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11/userdata/shm DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volumes/kubernetes.io~projected/kube-api-access-h7pqj DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22bdaed6b071b2a7f3b4f3bcae8b721e8093de017e762344e7d1284ba5363627/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba4999ae-bde0-4c8a-9652-4ea7811afdaa/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:352 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ade4a17684d37fefed46a10d44fe037fa3e31f03293a28edcb01190dc3ae3da/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac00a89862e2d7d36dd03849e2c6d54e9900e34dd709db7c348f4149ac42759c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b706a15-4789-4900-a62b-ba4275904124/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~projected/kube-api-access-nscr6 DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec32c8f1-eb5a-4a91-af80-f41d281d672e/volumes/kubernetes.io~projected/kube-api-access-smb44 DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9/volumes/kubernetes.io~projected/kube-api-access-7p5kq DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe97d3b6354bd4cba9b547d40d6af548bfebea221367f4dbe61edbc8713d7833/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f2c416d-69b2-49ef-a2e3-230c5935d67b/volumes/kubernetes.io~projected/kube-api-access-cbb8q DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19919dff-4a1d-470c-9d64-bedcb6bb5a2a/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67e87ecf6e5f06d23b7738041f77eb93f45b7d307c898cb183b310a0929e7ed7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9712dd02-a8c8-402f-bfcd-8e149dc4835b/volumes/kubernetes.io~projected/kube-api-access-zxbnk DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85/userdata/shm DeviceMajor:0 DeviceMinor:280 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:22bdaed6b071b2a MacAddress:1a:31:4a:6d:82:47 Speed:10000 Mtu:8901} {Name:67e87ecf6e5f06d MacAddress:0e:b9:12:25:db:69 Speed:10000 Mtu:8901} {Name:6b9c472d469007d MacAddress:f6:a4:2c:84:d1:34 Speed:10000 Mtu:8901} {Name:6c13b30394881c2 MacAddress:d6:55:8a:46:03:cf Speed:10000 Mtu:8901} {Name:8ade4a17684d37f MacAddress:c6:0c:74:eb:79:1c Speed:10000 Mtu:8901} {Name:b00aef3cb3d02d9 MacAddress:a6:c4:b0:0d:67:40 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:5e:bc:d9 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:15:2f:6d:17:6c Speed:0 Mtu:8901} {Name:c10bc50b15f6856 MacAddress:4e:03:9a:b5:83:8c Speed:10000 Mtu:8901} {Name:c4d09783ad02d9d MacAddress:8e:f2:0a:01:74:05 Speed:10000 Mtu:8901} {Name:c8b096ecebb97dd MacAddress:02:8a:dd:66:2d:78 Speed:10000 Mtu:8901} {Name:e8eb0f4b1ba2c23 MacAddress:3e:38:39:29:a0:ca Speed:10000 Mtu:8901} {Name:ee8eb3c35f76cdc MacAddress:ee:97:b3:76:7d:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:5e:bc:d9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6e:73:1b:1a:76:f1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d2:6c:93:e6:1c:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670444 6275 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670493 6275 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670784 6275 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670806 6275 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670936 6275 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670943 6275 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670953 6275 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.670972 6275 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671081 6275 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:29:31.671343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671313 6275 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:29:31.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671423 6275 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:29:31.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671458 6275 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:29:31.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671467 6275 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:29:31.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671476 6275 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:29:31.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671486 6275 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:29:31.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.671494 6275 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:29:31.672252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.672237 6275 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:29:31.672287 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.672255 6275 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:29:31.672788 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.672764 6275 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:29:31.681768 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.681752 6275 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:29:31.681768 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.681767 6275 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:29:31.681893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.681869 6275 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 01:29:31.682063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682053 6275 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:29:31.682094 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682082 6275 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:29:31.682194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682185 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:29:31.682238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682200 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:29:31.682238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682211 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:29:31.682238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682219 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:29:31.682238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682225 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:29:31.682238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682232 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:29:31.682238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682238 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682243 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682251 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682257 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682270 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682281 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682308 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:29:31.682501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.682315 6275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:29:31.685865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.685852 6275 server.go:1263] "Started kubelet" Oct 30 01:29:31.686016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.685936 6275 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:29:31.686445 ip-10-0-143-27 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:29:31.687280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.685994 6275 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:29:31.687280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.686873 6275 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:29:31.687280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.687206 6275 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:29:31.690169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.690046 6275 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:29:31.692674 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.692656 6275 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:29:31.692766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.692684 6275 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:29:31.692766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.692744 6275 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:21 +0000 UTC" deadline="2028-04-23 04:42:38.476648318 +0000 UTC" Oct 30 01:29:31.692911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.692764 6275 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12987h13m6.783887208s" Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.692798 6275 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693174 6275 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.692812 6275 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:31.692798 6275 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-27.ec2.internal\" not found" Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693417 6275 factory.go:55] Registering systemd factory Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693436 6275 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693737 6275 factory.go:153] Registering CRI-O factory Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693753 6275 factory.go:223] Registration of the crio container factory successfully Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693807 6275 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693824 6275 factory.go:103] Registering Raw factory Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.693838 6275 manager.go:1199] Started watching for new ooms in manager Oct 30 01:29:31.694510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.694217 6275 manager.go:319] Starting recovery of all containers Oct 30 01:29:31.695654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695615 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b9ca87-bcc5-4b7a-8517-db3484b7c5c8" volumeName="kubernetes.io/empty-dir/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-tmp-dir" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695669 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695683 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695698 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695733 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695749 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695764 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config" seLinuxMountContext="" Oct 30 01:29:31.695791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695787 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d69ca43-18c0-4010-87a9-7e3473dfe936" volumeName="kubernetes.io/secret/9d69ca43-18c0-4010-87a9-7e3473dfe936-cert" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695803 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" volumeName="kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-binary-copy" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695826 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e07d2c61-a1ab-4ece-8205-6515c8b8d880" volumeName="kubernetes.io/projected/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-api-access-7hf5v" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695848 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e07d2c61-a1ab-4ece-8205-6515c8b8d880" volumeName="kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695860 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" volumeName="kubernetes.io/projected/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-kube-api-access-h7pqj" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695881 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="467074bb-b3a4-444d-a8b9-9aae4f9d9d56" volumeName="kubernetes.io/configmap/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-iptables-alerter-script" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695913 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9712dd02-a8c8-402f-bfcd-8e149dc4835b" volumeName="kubernetes.io/projected/9712dd02-a8c8-402f-bfcd-8e149dc4835b-kube-api-access-zxbnk" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695928 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695941 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c859dda-f9c4-4c70-b6d2-e72b990dc465" volumeName="kubernetes.io/secret/9c859dda-f9c4-4c70-b6d2-e72b990dc465-agent-certs" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695965 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" volumeName="kubernetes.io/projected/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-kube-api-access-pt882" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.695995 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696037 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fcebb0-d866-476d-859e-f8b8626a4a79" volumeName="kubernetes.io/projected/76fcebb0-d866-476d-859e-f8b8626a4a79-kube-api-access-2mjk5" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696053 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603" volumeName="kubernetes.io/empty-dir/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-tmp-dir" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696066 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e07d2c61-a1ab-4ece-8205-6515c8b8d880" volumeName="kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696077 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696089 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fcebb0-d866-476d-859e-f8b8626a4a79" volumeName="kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696101 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2c416d-69b2-49ef-a2e3-230c5935d67b" volumeName="kubernetes.io/configmap/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696114 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603" volumeName="kubernetes.io/configmap/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-config-volume" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696190 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" volumeName="kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-script-lib" seLinuxMountContext="" Oct 30 01:29:31.696193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696204 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" volumeName="kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-env-overrides" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696215 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" volumeName="kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-config" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696236 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b9ca87-bcc5-4b7a-8517-db3484b7c5c8" volumeName="kubernetes.io/projected/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-kube-api-access-jg9x6" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696247 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d69ca43-18c0-4010-87a9-7e3473dfe936" volumeName="kubernetes.io/projected/9d69ca43-18c0-4010-87a9-7e3473dfe936-kube-api-access-8sbr9" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696261 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e07d2c61-a1ab-4ece-8205-6515c8b8d880" volumeName="kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696275 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="131e7807-d6f8-4104-836e-33c6dcbbc286" volumeName="kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696286 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141993bb-335c-4e9c-b08e-da41dd9d7f0f" volumeName="kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696300 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19919dff-4a1d-470c-9d64-bedcb6bb5a2a" volumeName="kubernetes.io/projected/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-kube-api-access-w24sk" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696344 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696359 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fcebb0-d866-476d-859e-f8b8626a4a79" volumeName="kubernetes.io/empty-dir/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-textfile" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696373 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec32c8f1-eb5a-4a91-af80-f41d281d672e" volumeName="kubernetes.io/projected/ec32c8f1-eb5a-4a91-af80-f41d281d672e-kube-api-access-smb44" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696389 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9" volumeName="kubernetes.io/projected/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-kube-api-access-7p5kq" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696401 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fcebb0-d866-476d-859e-f8b8626a4a79" volumeName="kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-tls" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696412 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b706a15-4789-4900-a62b-ba4275904124" volumeName="kubernetes.io/projected/7b706a15-4789-4900-a62b-ba4275904124-kube-api-access-j2c22" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696424 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696436 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b706a15-4789-4900-a62b-ba4275904124" volumeName="kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-etc-tuned" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696447 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b706a15-4789-4900-a62b-ba4275904124" volumeName="kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-tmp" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696461 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603" volumeName="kubernetes.io/secret/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-metrics-tls" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696472 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696487 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696512 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fcebb0-d866-476d-859e-f8b8626a4a79" volumeName="kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696526 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696543 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696556 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e07d2c61-a1ab-4ece-8205-6515c8b8d880" volumeName="kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696574 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19919dff-4a1d-470c-9d64-bedcb6bb5a2a" volumeName="kubernetes.io/secret/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696588 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2c416d-69b2-49ef-a2e3-230c5935d67b" volumeName="kubernetes.io/empty-dir/7f2c416d-69b2-49ef-a2e3-230c5935d67b-data-volume" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696601 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696615 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c859dda-f9c4-4c70-b6d2-e72b990dc465" volumeName="kubernetes.io/configmap/9c859dda-f9c4-4c70-b6d2-e72b990dc465-konnectivity-ca" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696629 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cded0e24-b028-447a-9bad-beda2dd20dbb" volumeName="kubernetes.io/projected/cded0e24-b028-447a-9bad-beda2dd20dbb-kube-api-access-zgv67" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696645 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9" volumeName="kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-daemon-config" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696668 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" volumeName="kubernetes.io/secret/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696682 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba4999ae-bde0-4c8a-9652-4ea7811afdaa" volumeName="kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696697 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" volumeName="kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696712 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" volumeName="kubernetes.io/projected/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-kube-api-access-nlrnd" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696725 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2c416d-69b2-49ef-a2e3-230c5935d67b" volumeName="kubernetes.io/projected/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-api-access-cbb8q" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696738 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" volumeName="kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696749 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e07d2c61-a1ab-4ece-8205-6515c8b8d880" volumeName="kubernetes.io/empty-dir/e07d2c61-a1ab-4ece-8205-6515c8b8d880-volume-directive-shadow" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696763 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="467074bb-b3a4-444d-a8b9-9aae4f9d9d56" volumeName="kubernetes.io/projected/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-kube-api-access-g7pmk" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696775 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696787 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fcebb0-d866-476d-859e-f8b8626a4a79" volumeName="kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696826 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2c416d-69b2-49ef-a2e3-230c5935d67b" volumeName="kubernetes.io/secret/7f2c416d-69b2-49ef-a2e3-230c5935d67b-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696844 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9712dd02-a8c8-402f-bfcd-8e149dc4835b" volumeName="kubernetes.io/configmap/9712dd02-a8c8-402f-bfcd-8e149dc4835b-serviceca" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696857 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696870 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603" volumeName="kubernetes.io/projected/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-kube-api-access-n8gmk" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696882 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec32c8f1-eb5a-4a91-af80-f41d281d672e" volumeName="kubernetes.io/empty-dir/ec32c8f1-eb5a-4a91-af80-f41d281d672e-tmp" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696895 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9" volumeName="kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cni-binary-copy" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696907 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e81463a-be01-4475-94b8-2a84429273b9" volumeName="kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696919 6275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b65ce43-3ae8-461b-b703-fbc0b37645ea" volumeName="kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca" seLinuxMountContext="" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696930 6275 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:29:31.697086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.696938 6275 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:29:31.716055 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.716025 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:29:31.720205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.720181 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:29:31.729107 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.729074 6275 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:29:31.729800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.729777 6275 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 01:29:31.730516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.730495 6275 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:29:31.730608 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.730524 6275 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:29:31.730608 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.730545 6275 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:29:31.730681 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:31.730584 6275 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 01:29:31.733080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.733058 6275 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:31.733280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.733206 6275 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:31.736957 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.736924 6275 generic.go:356] "Generic (PLEG): container finished" podID="148264b018741ac927337137ecf951d1" containerID="02eeddbc0faf83a6cec7d089651936247a2b5806955bef4f8fb4dfe140fe2338" exitCode=0 Oct 30 01:29:31.751232 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.751192 6275 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:31.751423 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.751390 6275 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:31.754170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.754142 6275 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="d50286d30562bbd539a955ddc5fa27c633ad0d574ab950dde1eae08198759f1b" exitCode=0 Oct 30 01:29:31.754170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.754166 6275 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="f8a02ac79be0f93f88c4b03b8ae462b83a349c2678440de10915fe1f49c2762a" exitCode=0 Oct 30 01:29:31.754273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.754176 6275 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="8fcb2e618fd648d310613e81229f83571eb1dfa23ec63ed68fb834475e590aa3" exitCode=0 Oct 30 01:29:31.754273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.754182 6275 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="1d9fcb617733900d422a8cd1dde865286b6189e612ee7e5c31465180056f017d" exitCode=0 Oct 30 01:29:31.754273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.754187 6275 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="eca6775891be01b1d265d26a39a80822296015fef82f27162503ba29e256750b" exitCode=0 Oct 30 01:29:31.754273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.754192 6275 generic.go:356] "Generic (PLEG): container finished" podID="d14d41c9-2491-4d02-83aa-356a6e8aa1e5" containerID="1241fe39afd5adfbc3a1b19129861c2661ddc432a76cbe25bd199dda9dd6c9dc" exitCode=0 Oct 30 01:29:31.760587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.760564 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:29:31.761464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.761420 6275 generic.go:356] "Generic (PLEG): container finished" podID="0f0245e1-cf5c-4d84-8072-bdd72bfe3f36" containerID="bfd791706e80bdc402c72e5544ec8e8c95afc96df2e52f43dab6157b59142873" exitCode=1 Oct 30 01:29:31.771028 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.771001 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" exitCode=0 Oct 30 01:29:31.781026 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.780998 6275 generic.go:356] "Generic (PLEG): container finished" podID="76fcebb0-d866-476d-859e-f8b8626a4a79" containerID="1f581c89e6e120b58134876113cf0d27aed7b14f9287297f2ec2d42ce37289bd" exitCode=0 Oct 30 01:29:31.797694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.797672 6275 manager.go:324] Recovery completed Oct 30 01:29:31.820260 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820238 6275 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:29:31.820260 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820254 6275 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:29:31.820419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820271 6275 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:29:31.820419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820407 6275 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 01:29:31.820478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820418 6275 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 01:29:31.820478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820449 6275 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 01:29:31.820478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820457 6275 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 01:29:31.820478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820465 6275 policy_none.go:50] "Start" Oct 30 01:29:31.820478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820474 6275 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:29:31.820612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820484 6275 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:29:31.820640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820617 6275 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 01:29:31.820640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820625 6275 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 01:29:31.820640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.820631 6275 policy_none.go:44] "Start" Oct 30 01:29:31.825653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.825094 6275 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:29:31.830708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.830692 6275 manager.go:341] "Starting Device Plugin manager" Oct 30 01:29:31.830794 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:31.830719 6275 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:29:31.830794 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.830727 6275 server.go:87] "Starting device plugin registration server" Oct 30 01:29:31.830880 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:31.830797 6275 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 01:29:31.831051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831038 6275 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:29:31.831088 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831057 6275 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:29:31.831174 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831157 6275 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:29:31.831283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831250 6275 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:29:31.831362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831337 6275 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:29:31.831362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831345 6275 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:29:31.831515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.831401 6275 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:29:31.831281472Z","UUID":"fca16212-54fc-4a0c-a24e-48c0a156b662","Handler":null,"Name":"","Endpoint":""} Oct 30 01:29:31.832839 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.832822 6275 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:29:31.832945 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:31.832841 6275 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:29:31.832945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.832849 6275 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:29:31.931801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.931756 6275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:29:31.934004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.933981 6275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:29:31.934104 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.934019 6275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:29:31.934104 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.934034 6275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-27.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:29:31.934104 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.934084 6275 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-27.ec2.internal" Oct 30 01:29:31.972448 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.972420 6275 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-27.ec2.internal" Oct 30 01:29:31.972549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:31.972510 6275 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.030947 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.030900 6275 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal"] Oct 30 01:29:32.031539 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031521 6275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.031539 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031517 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" event={"ID":"148264b018741ac927337137ecf951d1","Type":"ContainerStarted","Data":"ddf48252da9806db38e38ca18513d49203be5ed3c21396657c454fdba2212e85"} Oct 30 01:29:32.031699 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031573 6275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.031699 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031577 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" event={"ID":"148264b018741ac927337137ecf951d1","Type":"ContainerDied","Data":"02eeddbc0faf83a6cec7d089651936247a2b5806955bef4f8fb4dfe140fe2338"} Oct 30 01:29:32.031699 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031603 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" event={"ID":"148264b018741ac927337137ecf951d1","Type":"ContainerStarted","Data":"15b268052f8bac8534615d970e2069100ea0b435d1f4ac94abad30ca3d38e018"} Oct 30 01:29:32.031835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031733 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" event={"ID":"0fc72893a455f4d88649494269f87258","Type":"ContainerStarted","Data":"b327a9025af5d9a64688ec3ef42ce7ca8b991e7a6726a2f9ab290170e3dcc499"} Oct 30 01:29:32.031835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.031755 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" event={"ID":"0fc72893a455f4d88649494269f87258","Type":"ContainerStarted","Data":"766567a69e7b7f5e7d82ee5323d3ae59e6c67404aefeb308964e06e2bb5d0f11"} Oct 30 01:29:32.052157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.052141 6275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:32.052213 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:32.052178 6275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.052642 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.052625 6275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:32.052725 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:32.052667 6275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.132074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.131976 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.132074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.132020 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.132074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.132049 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fc72893a455f4d88649494269f87258-config\") pod \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" (UID: \"0fc72893a455f4d88649494269f87258\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.232588 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.232552 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.232745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.232592 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.232745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.232618 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fc72893a455f4d88649494269f87258-config\") pod \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" (UID: \"0fc72893a455f4d88649494269f87258\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.232745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.232662 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fc72893a455f4d88649494269f87258-config\") pod \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" (UID: \"0fc72893a455f4d88649494269f87258\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.232745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.232663 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.232745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.232663 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/148264b018741ac927337137ecf951d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" (UID: \"148264b018741ac927337137ecf951d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.671811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.671773 6275 apiserver.go:52] "Watching apiserver" Oct 30 01:29:32.718045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.718018 6275 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:29:32.718601 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.718580 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xv85r","openshift-console/console-545767f87d-ccnvd","openshift-console/downloads-6dd789c8b7-nc6vm","openshift-dns/node-resolver-5w55t","openshift-monitoring/alertmanager-main-0","openshift-network-diagnostics/network-check-target-9qlzc","openshift-cluster-node-tuning-operator/tuned-s98cl","openshift-image-registry/node-ca-qmg28","openshift-ovn-kubernetes/ovnkube-node-2kd25","kube-system/global-pull-secret-syncer-jt6kw","openshift-multus/multus-gcqqc","kube-system/konnectivity-agent-lwf9z","openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv","openshift-insights/insights-runtime-extractor-zdlwv","openshift-multus/multus-additional-cni-plugins-f9pb2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf","openshift-monitoring/node-exporter-8mp4k","openshift-network-operator/iptables-alerter-7c4wq","kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal","openshift-ingress-canary/ingress-canary-gm2cs","openshift-dns/dns-default-9gx4n"] Oct 30 01:29:32.734788 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.734763 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:29:32.734935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.734807 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:29:32.735026 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.735008 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.735080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.735067 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.735877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.735855 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.735916 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.735907 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:29:32.736075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.736063 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:29:32.739142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.739111 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:29:32.739571 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.739556 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.739621 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.739571 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.739621 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.739582 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.739863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.739851 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.740078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740065 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:29:32.740078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740071 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.740165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740086 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.740390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740366 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.740390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740367 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.740525 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740400 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:29:32.740525 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.740418 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.742694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.742679 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:29:32.742977 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.742961 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:29:32.742977 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.742971 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lm6wv\"" Oct 30 01:29:32.743086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.742974 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglfc\"" Oct 30 01:29:32.743086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.742993 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:29:32.743086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.742969 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c6clm\"" Oct 30 01:29:32.743086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.743019 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7lll\"" Oct 30 01:29:32.743086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.743024 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:29:32.743086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.743021 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkdgr\"" Oct 30 01:29:32.743308 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.743203 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:29:32.743740 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.743727 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lzbd2\"" Oct 30 01:29:32.743784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.743764 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:29:32.748583 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.748569 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.749424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.749411 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7p9xn\"" Oct 30 01:29:32.754874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.754854 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:29:32.754874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.754870 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.754994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.754864 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:29:32.754994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.754875 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.754994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.754871 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:29:32.754994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.754906 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.755152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.755054 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rwcx9\"" Oct 30 01:29:32.755152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.755074 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:29:32.755152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.755080 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2gfdm\"" Oct 30 01:29:32.755152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.755100 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:29:32.756911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.756895 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:29:32.756968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.756927 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:29:32.757995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.757983 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.758577 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.758561 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.758628 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.758563 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.758628 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.758594 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.758764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.758752 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 01:29:32.758820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.758779 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 01:29:32.758820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.758809 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:29:32.759013 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759000 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:29:32.759074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759055 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:29:32.759307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759292 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 01:29:32.759365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759321 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 01:29:32.759365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759327 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.759365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759341 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.759365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.759359 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 01:29:32.760563 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.760547 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 01:29:32.760659 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.760646 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zqs4x\"" Oct 30 01:29:32.760697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.760670 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfcpg\"" Oct 30 01:29:32.760697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.760675 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 01:29:32.761108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.761095 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:29:32.761373 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.761358 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 01:29:32.761470 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.761379 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 01:29:32.762198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762187 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 01:29:32.762609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762592 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 01:29:32.762609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762598 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kg7c7\"" Oct 30 01:29:32.762751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762641 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 01:29:32.762751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762647 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 01:29:32.762751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762619 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 01:29:32.762751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762672 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 01:29:32.762751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762719 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 01:29:32.762751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762731 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 01:29:32.762930 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762768 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:29:32.762930 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762771 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:29:32.762930 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762815 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fskhp\"" Oct 30 01:29:32.762930 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.762867 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:29:32.765558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765543 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:29:32.765819 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765794 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2p848\"" Oct 30 01:29:32.765819 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765810 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tntm7\"" Oct 30 01:29:32.765819 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765814 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qp94x\"" Oct 30 01:29:32.765950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765796 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f9jp6\"" Oct 30 01:29:32.765950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765828 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 01:29:32.765950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765838 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bfmqc\"" Oct 30 01:29:32.765950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.765866 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:29:32.790962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.790939 6275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.791072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.791059 6275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.795703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.795684 6275 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:29:32.819080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.819059 6275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:32.819080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.819058 6275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:32.819271 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:32.819102 6275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-27.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.819271 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:32.819164 6275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-27.ec2.internal" Oct 30 01:29:32.834892 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.834869 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9x6\" (UniqueName: \"kubernetes.io/projected/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-kube-api-access-jg9x6\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.834990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.834904 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysconfig\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.834990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.834926 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-systemd\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.835117 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835095 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.835260 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835244 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.835293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835272 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.835337 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835292 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-netns\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.835337 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835308 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-device-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.835429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835343 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.835429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835384 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbnk\" (UniqueName: \"kubernetes.io/projected/9712dd02-a8c8-402f-bfcd-8e149dc4835b-kube-api-access-zxbnk\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.835527 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835483 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.835574 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835536 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-config-volume\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.835627 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835574 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.835627 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835590 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c859dda-f9c4-4c70-b6d2-e72b990dc465-agent-certs\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:29:32.835726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835649 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgv67\" (UniqueName: \"kubernetes.io/projected/cded0e24-b028-447a-9bad-beda2dd20dbb-kube-api-access-zgv67\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.835773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835722 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-systemd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.835773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835753 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.835877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835779 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-bin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.835877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835800 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-config-volume\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.835877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835838 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c859dda-f9c4-4c70-b6d2-e72b990dc465-agent-certs\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:29:32.835877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835849 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.835877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835859 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835888 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835913 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c859dda-f9c4-4c70-b6d2-e72b990dc465-konnectivity-ca\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835928 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-socket-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835943 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-multus-certs\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.835985 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-systemd-units\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836005 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-netd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836021 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836060 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-etc-kubernetes\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.836102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836092 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-textfile\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836117 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836155 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-kubernetes\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836172 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836176 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c859dda-f9c4-4c70-b6d2-e72b990dc465-konnectivity-ca\") pod \"konnectivity-agent-lwf9z\" (UID: \"9c859dda-f9c4-4c70-b6d2-e72b990dc465\") " pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836188 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836191 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-textfile\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836205 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-socket-dir-parent\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836226 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cnibin\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836262 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-metrics-tls\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836278 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836293 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-tls\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836312 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-sys\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836338 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-var-lib-kubelet\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836384 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836392 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836421 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836448 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.836471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836471 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-tmp-dir\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836521 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-metrics-tls\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836572 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d69ca43-18c0-4010-87a9-7e3473dfe936-cert\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836581 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/141993bb-335c-4e9c-b08e-da41dd9d7f0f-original-pull-secret\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836602 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836626 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-tls\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836632 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836628 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-metrics-certs\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836633 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-tmp-dir\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836639 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836786 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836788 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovn-node-metrics-cert\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836797 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d69ca43-18c0-4010-87a9-7e3473dfe936-cert\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836812 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-registration-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836830 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836848 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9712dd02-a8c8-402f-bfcd-8e149dc4835b-serviceca\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836882 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-wtmp\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.836953 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hf5v\" (UniqueName: \"kubernetes.io/projected/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-api-access-7hf5v\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837061 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5kq\" (UniqueName: \"kubernetes.io/projected/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-kube-api-access-7p5kq\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837101 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovn-node-metrics-cert\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837149 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837183 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837194 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837231 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-script-lib\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837283 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-multus\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837311 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837372 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837447 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-script-lib\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837442 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837498 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837509 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-run\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837536 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt882\" (UniqueName: \"kubernetes.io/projected/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-kube-api-access-pt882\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837556 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-system-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837561 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9712dd02-a8c8-402f-bfcd-8e149dc4835b-serviceca\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837577 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837618 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-kubelet\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837656 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837735 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837770 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837806 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-log-socket\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837839 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-host-slash\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837895 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f2c416d-69b2-49ef-a2e3-230c5935d67b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837928 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837953 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837955 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.837981 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gmk\" (UniqueName: \"kubernetes.io/projected/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-kube-api-access-n8gmk\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.837995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838005 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-sys\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838026 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pmk\" (UniqueName: \"kubernetes.io/projected/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-kube-api-access-g7pmk\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838042 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838063 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-netns\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838097 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838085 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f2c416d-69b2-49ef-a2e3-230c5935d67b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838137 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-sys-fs\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838200 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f2c416d-69b2-49ef-a2e3-230c5935d67b-crio-socket\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838248 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2c22\" (UniqueName: \"kubernetes.io/projected/7b706a15-4789-4900-a62b-ba4275904124-kube-api-access-j2c22\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838283 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838301 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838336 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24sk\" (UniqueName: \"kubernetes.io/projected/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-kube-api-access-w24sk\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838342 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838342 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838415 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-root\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838424 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838513 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-conf\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838552 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-lib-modules\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838576 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838602 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-bin\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838623 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838639 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-var-lib-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838711 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-ovn\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838799 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbb8q\" (UniqueName: \"kubernetes.io/projected/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-api-access-cbb8q\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838819 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e07d2c61-a1ab-4ece-8205-6515c8b8d880-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838830 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838864 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838890 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e07d2c61-a1ab-4ece-8205-6515c8b8d880-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.838894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838898 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.838993 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839035 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cni-binary-copy\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839040 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839161 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec32c8f1-eb5a-4a91-af80-f41d281d672e-tmp\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839170 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07d2c61-a1ab-4ece-8205-6515c8b8d880-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839241 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbr9\" (UniqueName: \"kubernetes.io/projected/9d69ca43-18c0-4010-87a9-7e3473dfe936-kube-api-access-8sbr9\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839288 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cni-binary-copy\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839297 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec32c8f1-eb5a-4a91-af80-f41d281d672e-tmp\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839328 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-conf-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839414 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-daemon-config\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839433 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-hosts-file\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839450 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9712dd02-a8c8-402f-bfcd-8e149dc4835b-host\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839481 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrnd\" (UniqueName: \"kubernetes.io/projected/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-kube-api-access-nlrnd\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839501 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smb44\" (UniqueName: \"kubernetes.io/projected/ec32c8f1-eb5a-4a91-af80-f41d281d672e-kube-api-access-smb44\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839537 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-etc-tuned\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839587 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-daemon-config\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839608 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-etc-tuned\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839626 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-kubelet-config\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.839652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839658 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-dbus\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839682 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-hostroot\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839748 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839787 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-env-overrides\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839803 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839821 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839847 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839923 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839940 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839966 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-node-log\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839982 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-env-overrides\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.839991 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840001 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-config\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840018 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840028 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-tmp\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840070 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-slash\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840096 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-etc-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840112 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b706a15-4789-4900-a62b-ba4275904124-tmp\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840179 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pqj\" (UniqueName: \"kubernetes.io/projected/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-kube-api-access-h7pqj\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840195 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-ovnkube-config\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840208 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cnibin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840239 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-metrics-client-ca\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840294 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-tmp-dir\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840324 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjk5\" (UniqueName: \"kubernetes.io/projected/76fcebb0-d866-476d-859e-f8b8626a4a79-kube-api-access-2mjk5\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840349 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-kubelet\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840358 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-tmp-dir\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.840417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840382 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76fcebb0-d866-476d-859e-f8b8626a4a79-metrics-client-ca\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840506 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840529 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-iptables-alerter-script\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840545 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f2c416d-69b2-49ef-a2e3-230c5935d67b-data-volume\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840566 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840599 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840621 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f2c416d-69b2-49ef-a2e3-230c5935d67b-data-volume\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840625 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-system-cni-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840646 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840652 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840679 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-modprobe-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840707 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-host\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840722 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-iptables-alerter-script\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840732 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840827 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840846 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-k8s-cni-cncf-io\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840873 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.841372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.840902 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.842104 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.841776 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.854884 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.854863 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9x6\" (UniqueName: \"kubernetes.io/projected/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-kube-api-access-jg9x6\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.858590 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.858563 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbnk\" (UniqueName: \"kubernetes.io/projected/9712dd02-a8c8-402f-bfcd-8e149dc4835b-kube-api-access-zxbnk\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.858718 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.858639 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgv67\" (UniqueName: \"kubernetes.io/projected/cded0e24-b028-447a-9bad-beda2dd20dbb-kube-api-access-zgv67\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.859101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.859081 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") pod \"console-545767f87d-ccnvd\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:32.859222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.859205 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hf5v\" (UniqueName: \"kubernetes.io/projected/e07d2c61-a1ab-4ece-8205-6515c8b8d880-kube-api-access-7hf5v\") pod \"kube-state-metrics-5dc8d7d89b-nfpnf\" (UID: \"e07d2c61-a1ab-4ece-8205-6515c8b8d880\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-nfpnf" Oct 30 01:29:32.859281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.859258 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt882\" (UniqueName: \"kubernetes.io/projected/ba4999ae-bde0-4c8a-9652-4ea7811afdaa-kube-api-access-pt882\") pod \"network-metrics-daemon-xv85r\" (UID: \"ba4999ae-bde0-4c8a-9652-4ea7811afdaa\") " pod="openshift-multus/network-metrics-daemon-xv85r" Oct 30 01:29:32.859741 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.859724 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5kq\" (UniqueName: \"kubernetes.io/projected/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-kube-api-access-7p5kq\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.865653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.865633 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjk5\" (UniqueName: \"kubernetes.io/projected/76fcebb0-d866-476d-859e-f8b8626a4a79-kube-api-access-2mjk5\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.865826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.865810 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gmk\" (UniqueName: \"kubernetes.io/projected/9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603-kube-api-access-n8gmk\") pod \"dns-default-9gx4n\" (UID: \"9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603\") " pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:32.871306 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.871290 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbb8q\" (UniqueName: \"kubernetes.io/projected/7f2c416d-69b2-49ef-a2e3-230c5935d67b-kube-api-access-cbb8q\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.871559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.871543 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24sk\" (UniqueName: \"kubernetes.io/projected/19919dff-4a1d-470c-9d64-bedcb6bb5a2a-kube-api-access-w24sk\") pod \"monitoring-plugin-77f4f76594-5c7nv\" (UID: \"19919dff-4a1d-470c-9d64-bedcb6bb5a2a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:32.871622 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.871592 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") pod \"alertmanager-main-0\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:32.871753 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.871739 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2c22\" (UniqueName: \"kubernetes.io/projected/7b706a15-4789-4900-a62b-ba4275904124-kube-api-access-j2c22\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.872530 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.872509 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smb44\" (UniqueName: \"kubernetes.io/projected/ec32c8f1-eb5a-4a91-af80-f41d281d672e-kube-api-access-smb44\") pod \"downloads-6dd789c8b7-nc6vm\" (UID: \"ec32c8f1-eb5a-4a91-af80-f41d281d672e\") " pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:32.876019 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.876003 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pqj\" (UniqueName: \"kubernetes.io/projected/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-kube-api-access-h7pqj\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.876558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.876538 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pmk\" (UniqueName: \"kubernetes.io/projected/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-kube-api-access-g7pmk\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.876626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.876592 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrnd\" (UniqueName: \"kubernetes.io/projected/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-kube-api-access-nlrnd\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.876964 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.876949 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgf5g\" (UniqueName: \"kubernetes.io/projected/131e7807-d6f8-4104-836e-33c6dcbbc286-kube-api-access-sgf5g\") pod \"network-check-target-9qlzc\" (UID: \"131e7807-d6f8-4104-836e-33c6dcbbc286\") " pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:32.876996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.876970 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbr9\" (UniqueName: \"kubernetes.io/projected/9d69ca43-18c0-4010-87a9-7e3473dfe936-kube-api-access-8sbr9\") pod \"ingress-canary-gm2cs\" (UID: \"9d69ca43-18c0-4010-87a9-7e3473dfe936\") " pod="openshift-ingress-canary/ingress-canary-gm2cs" Oct 30 01:29:32.941294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941230 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-k8s-cni-cncf-io\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.941294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941258 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysconfig\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.941294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941274 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-systemd\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.941294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941290 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941325 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-netns\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941361 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysconfig\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941367 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-device-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941397 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-k8s-cni-cncf-io\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941404 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941373 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-netns\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941371 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-systemd\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941443 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-systemd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941407 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-systemd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941464 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-device-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941480 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941498 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-bin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941522 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.941531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941538 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-socket-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941544 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941553 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-multus-certs\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941564 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-bin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941582 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-systemd-units\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941604 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-multus-certs\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941609 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-netd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941613 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-socket-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941633 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941642 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941654 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-netd\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941664 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-etc-kubernetes\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941683 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-kubernetes\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941694 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-etc-kubernetes\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941640 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-systemd-units\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941698 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-socket-dir-parent\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941719 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-kubernetes\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941728 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-socket-dir-parent\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941681 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941738 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cnibin\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941761 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-sys\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941775 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-var-lib-kubelet\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941792 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941813 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-registration-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941839 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-wtmp\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941846 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941840 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-var-lib-kubelet\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941865 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941879 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-registration-dir\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941885 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-node-exporter-wtmp\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941890 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-multus\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941884 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-sys\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941790 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-cnibin\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941913 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-run\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941929 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-system-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941944 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-kubelet\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941943 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-cni-multus\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941887 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941961 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-log-socket\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941990 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-log-socket\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941993 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-var-lib-kubelet\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942001 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-system-cni-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941967 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-run\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.941996 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-host-slash\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942029 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/467074bb-b3a4-444d-a8b9-9aae4f9d9d56-host-slash\") pod \"iptables-alerter-7c4wq\" (UID: \"467074bb-b3a4-444d-a8b9-9aae4f9d9d56\") " pod="openshift-network-operator/iptables-alerter-7c4wq" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942039 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942065 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-sys\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942095 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-sys\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942067 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942095 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-netns\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942141 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-host-run-netns\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.942097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942158 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-sys-fs\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942173 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f2c416d-69b2-49ef-a2e3-230c5935d67b-crio-socket\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942191 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-root\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942203 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-conf\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942219 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-lib-modules\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942237 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-bin\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942255 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-var-lib-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942256 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f2c416d-69b2-49ef-a2e3-230c5935d67b-crio-socket\") pod \"insights-runtime-extractor-zdlwv\" (UID: \"7f2c416d-69b2-49ef-a2e3-230c5935d67b\") " pod="openshift-insights/insights-runtime-extractor-zdlwv" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942254 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76fcebb0-d866-476d-859e-f8b8626a4a79-root\") pod \"node-exporter-8mp4k\" (UID: \"76fcebb0-d866-476d-859e-f8b8626a4a79\") " pod="openshift-monitoring/node-exporter-8mp4k" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942284 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-ovn\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942292 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-cni-bin\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942294 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cded0e24-b028-447a-9bad-beda2dd20dbb-sys-fs\") pod \"aws-ebs-csi-driver-node-xsgjt\" (UID: \"cded0e24-b028-447a-9bad-beda2dd20dbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsgjt" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942315 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-conf-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942319 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-var-lib-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942332 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-hosts-file\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942334 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-lib-modules\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942342 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-run-ovn\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942350 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9712dd02-a8c8-402f-bfcd-8e149dc4835b-host\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942360 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-multus-conf-dir\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942364 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-sysctl-conf\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942375 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-kubelet-config\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942402 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-dbus\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942399 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58b9ca87-bcc5-4b7a-8517-db3484b7c5c8-hosts-file\") pod \"node-resolver-5w55t\" (UID: \"58b9ca87-bcc5-4b7a-8517-db3484b7c5c8\") " pod="openshift-dns/node-resolver-5w55t" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942423 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-hostroot\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942441 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9712dd02-a8c8-402f-bfcd-8e149dc4835b-host\") pod \"node-ca-qmg28\" (UID: \"9712dd02-a8c8-402f-bfcd-8e149dc4835b\") " pod="openshift-image-registry/node-ca-qmg28" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942445 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-node-log\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942450 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-kubelet-config\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942463 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-slash\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942472 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/141993bb-335c-4e9c-b08e-da41dd9d7f0f-dbus\") pod \"global-pull-secret-syncer-jt6kw\" (UID: \"141993bb-335c-4e9c-b08e-da41dd9d7f0f\") " pod="kube-system/global-pull-secret-syncer-jt6kw" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942477 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-etc-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942480 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-hostroot\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942490 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-node-log\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942506 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-etc-openvswitch\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942515 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-slash\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942508 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cnibin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942535 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9-cnibin\") pod \"multus-gcqqc\" (UID: \"0c6eff6e-ebcf-45c0-a37f-3cd99bc787e9\") " pod="openshift-multus/multus-gcqqc" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942550 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-kubelet\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942569 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942587 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-kubelet\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942588 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-system-cni-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942609 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-modprobe-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942613 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f0245e1-cf5c-4d84-8072-bdd72bfe3f36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kd25\" (UID: \"0f0245e1-cf5c-4d84-8072-bdd72bfe3f36\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942625 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-host\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942637 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d14d41c9-2491-4d02-83aa-356a6e8aa1e5-system-cni-dir\") pod \"multus-additional-cni-plugins-f9pb2\" (UID: \"d14d41c9-2491-4d02-83aa-356a6e8aa1e5\") " pod="openshift-multus/multus-additional-cni-plugins-f9pb2" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942668 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-host\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:32.943638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:32.942703 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b706a15-4789-4900-a62b-ba4275904124-etc-modprobe-d\") pod \"tuned-s98cl\" (UID: \"7b706a15-4789-4900-a62b-ba4275904124\") " pod="openshift-cluster-node-tuning-operator/tuned-s98cl" Oct 30 01:29:33.032578 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:33.032558 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:29:34.884103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:34.884056 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9gx4n" Oct 30 01:29:35.105348 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:35.105316 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:29:35.887233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:35.887194 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:35.891334 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:35.891309 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:36.545973 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:36.545940 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5c7nv" Oct 30 01:29:38.446607 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.446571 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d7d4bbb89-6hl5t"] Oct 30 01:29:38.517611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.517581 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7d4bbb89-6hl5t"] Oct 30 01:29:38.517796 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.517714 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.564653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.564617 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 01:29:38.596035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.595992 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596214 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596043 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596214 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596074 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596214 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596106 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596261 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596351 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596396 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7l6\" (UniqueName: \"kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.596493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.596444 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697702 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697626 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697702 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697680 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697710 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697730 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697753 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697802 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697847 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7l6\" (UniqueName: \"kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.697913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.697886 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.698413 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.698385 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.698660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.698624 6275 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:29:38.698943 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.698914 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.703572 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.699235 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.703572 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.699542 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.703572 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.700248 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.703572 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.701376 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.704495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.704470 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.720892 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.720867 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7l6\" (UniqueName: \"kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6\") pod \"console-d7d4bbb89-6hl5t\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.847018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.846983 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:38.862735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:38.862699 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:39.172035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:39.171994 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7d4bbb89-6hl5t"] Oct 30 01:29:39.176037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:39.176015 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:29:39.177075 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:39.177051 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9c23ee_3b0a_406e_8f9b_46b56e98854b.slice/crio-2d4e02586d71a45740c8760d35197db0d3b9d26facb9197da1e50aa67ab2d672 WatchSource:0}: Error finding container 2d4e02586d71a45740c8760d35197db0d3b9d26facb9197da1e50aa67ab2d672: Status 404 returned error can't find the container with id 2d4e02586d71a45740c8760d35197db0d3b9d26facb9197da1e50aa67ab2d672 Oct 30 01:29:39.828758 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:39.828723 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d4bbb89-6hl5t" event={"ID":"ef9c23ee-3b0a-406e-8f9b-46b56e98854b","Type":"ContainerStarted","Data":"6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f"} Oct 30 01:29:39.829270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:39.828783 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d4bbb89-6hl5t" event={"ID":"ef9c23ee-3b0a-406e-8f9b-46b56e98854b","Type":"ContainerStarted","Data":"2d4e02586d71a45740c8760d35197db0d3b9d26facb9197da1e50aa67ab2d672"} Oct 30 01:29:39.869341 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:39.869292 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d7d4bbb89-6hl5t" podStartSLOduration=1.869273827 podStartE2EDuration="1.869273827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:39.868996171 +0000 UTC m=+8.260803804" watchObservedRunningTime="2026-10-30 01:29:39.869273827 +0000 UTC m=+8.261081461" Oct 30 01:29:40.206906 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.206829 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:40.207281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.207249 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="alertmanager" containerID="cri-o://7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" gracePeriod=120 Oct 30 01:29:40.207398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.207290 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy" containerID="cri-o://d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" gracePeriod=120 Oct 30 01:29:40.207398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.207313 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="prom-label-proxy" containerID="cri-o://ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" gracePeriod=120 Oct 30 01:29:40.207474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.207397 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="config-reloader" containerID="cri-o://e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" gracePeriod=120 Oct 30 01:29:40.207474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.207263 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-web" containerID="cri-o://141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" gracePeriod=120 Oct 30 01:29:40.207474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.207324 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-metric" containerID="cri-o://c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" gracePeriod=120 Oct 30 01:29:40.503461 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.503428 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:40.618633 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618591 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.618801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618636 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.618801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618676 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.618801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618704 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.618801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618736 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.618801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618778 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618815 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618850 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618901 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618931 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.618982 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.619021 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.619063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.619052 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") pod \"6e81463a-be01-4475-94b8-2a84429273b9\" (UID: \"6e81463a-be01-4475-94b8-2a84429273b9\") " Oct 30 01:29:40.626406 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.626331 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:40.626840 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.626794 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:40.627621 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.627576 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:40.629816 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.629788 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.630707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.630659 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "kube-api-access-nscr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:40.631666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.631639 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.631828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.631797 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:40.631950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.631822 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.632024 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.631986 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.632148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.632111 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.632902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.632874 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.633706 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.633686 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:40.634979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.634958 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config" pod "6e81463a-be01-4475-94b8-2a84429273b9" (UID: "6e81463a-be01-4475-94b8-2a84429273b9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:40.719938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.719905 6275 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-main-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.719941 6275 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.719957 6275 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.719972 6275 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-tls-assets\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.719985 6275 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-web-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.719998 6275 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-main-db\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720012 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscr6\" (UniqueName: \"kubernetes.io/projected/6e81463a-be01-4475-94b8-2a84429273b9-kube-api-access-nscr6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720036 6275 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720049 6275 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720061 6275 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e81463a-be01-4475-94b8-2a84429273b9-cluster-tls-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720470 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720074 6275 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720470 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720088 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e81463a-be01-4475-94b8-2a84429273b9-metrics-client-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.720470 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.720102 6275 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e81463a-be01-4475-94b8-2a84429273b9-config-out\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.837297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837260 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" exitCode=0 Oct 30 01:29:40.837297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837289 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" exitCode=0 Oct 30 01:29:40.837297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837298 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" exitCode=0 Oct 30 01:29:40.837297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837304 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" exitCode=0 Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837310 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" exitCode=0 Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837315 6275 generic.go:356] "Generic (PLEG): container finished" podID="6e81463a-be01-4475-94b8-2a84429273b9" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" exitCode=0 Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837356 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837288 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837449 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837493 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837534 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837574 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837534 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837625 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} Oct 30 01:29:40.837925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.837663 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e81463a-be01-4475-94b8-2a84429273b9","Type":"ContainerDied","Data":"e8eb0f4b1ba2c23567e265501d9eb5fcdbcc7db22c7e297afd8ca1e550d8ae85"} Oct 30 01:29:40.851046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.851030 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.858609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.858535 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.866322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.866303 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.882680 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.882662 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.888046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.888025 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:40.894763 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.894747 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.898926 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.898905 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:40.903018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.902996 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.911187 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.911167 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.911496 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.911463 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.911560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.911501 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} err="failed to get container status \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" Oct 30 01:29:40.911560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.911545 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.911902 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.911872 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.911979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.911911 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} err="failed to get container status \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" Oct 30 01:29:40.911979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.911934 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.912338 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.912322 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.912410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.912342 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} err="failed to get container status \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" Oct 30 01:29:40.912410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.912356 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.912627 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.912607 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.912679 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.912631 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} err="failed to get container status \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" Oct 30 01:29:40.912679 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.912650 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.912972 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.912935 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.913070 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.912979 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} err="failed to get container status \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" Oct 30 01:29:40.913070 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913001 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.913307 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.913254 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.913307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913302 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} err="failed to get container status \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" Oct 30 01:29:40.913399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913316 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.913613 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:40.913594 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.913701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913622 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} err="failed to get container status \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" Oct 30 01:29:40.913701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913640 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.913940 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913915 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} err="failed to get container status \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" Oct 30 01:29:40.914023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.913940 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.914241 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.914223 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} err="failed to get container status \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" Oct 30 01:29:40.914307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.914241 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.914534 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.914510 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} err="failed to get container status \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" Oct 30 01:29:40.914583 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.914537 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.914788 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.914769 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} err="failed to get container status \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" Oct 30 01:29:40.914788 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.914786 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.915093 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915067 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} err="failed to get container status \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" Oct 30 01:29:40.915093 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915093 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.915372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915344 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} err="failed to get container status \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" Oct 30 01:29:40.915445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915372 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.915709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915692 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} err="failed to get container status \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" Oct 30 01:29:40.915773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915712 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.915995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915976 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} err="failed to get container status \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" Oct 30 01:29:40.916059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.915993 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.916282 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.916261 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} err="failed to get container status \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" Oct 30 01:29:40.916361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.916282 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.916562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.916543 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} err="failed to get container status \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" Oct 30 01:29:40.916644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.916563 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.916820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.916794 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} err="failed to get container status \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" Oct 30 01:29:40.916888 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.916821 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.917049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917025 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} err="failed to get container status \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" Oct 30 01:29:40.917049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917045 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.917339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917317 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} err="failed to get container status \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" Oct 30 01:29:40.917339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917338 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.917620 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917593 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} err="failed to get container status \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" Oct 30 01:29:40.917703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917622 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.917993 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917970 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} err="failed to get container status \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" Oct 30 01:29:40.918065 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.917994 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.918359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.918325 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} err="failed to get container status \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" Oct 30 01:29:40.918459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.918360 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.918662 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.918642 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} err="failed to get container status \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" Oct 30 01:29:40.918662 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.918660 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.918971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.918945 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} err="failed to get container status \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" Oct 30 01:29:40.919059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.918972 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.919279 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919248 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} err="failed to get container status \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" Oct 30 01:29:40.919279 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919268 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.919517 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919499 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} err="failed to get container status \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" Oct 30 01:29:40.919563 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919517 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.919754 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919733 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} err="failed to get container status \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" Oct 30 01:29:40.919754 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919754 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.919994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919974 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} err="failed to get container status \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" Oct 30 01:29:40.919994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.919993 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.920262 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.920246 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} err="failed to get container status \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" Oct 30 01:29:40.920262 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.920262 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.920512 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.920489 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} err="failed to get container status \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" Oct 30 01:29:40.920571 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.920513 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.920790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.920759 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} err="failed to get container status \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" Oct 30 01:29:40.920790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.920789 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.921269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.921237 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} err="failed to get container status \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" Oct 30 01:29:40.921336 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.921270 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.921606 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.921586 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} err="failed to get container status \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" Oct 30 01:29:40.921681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.921608 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.921932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.921831 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} err="failed to get container status \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" Oct 30 01:29:40.921932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.921855 6275 scope.go:122] "RemoveContainer" containerID="ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.922447 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea"} err="failed to get container status \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": rpc error: code = NotFound desc = could not find container \"ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea\": container with ID starting with ec18537d2076ccd648f4599f3afe5b1bf91e5c09b8550d441bc11f00a24e09ea not found: ID does not exist" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.922476 6275 scope.go:122] "RemoveContainer" containerID="c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.923278 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f"} err="failed to get container status \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": rpc error: code = NotFound desc = could not find container \"c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f\": container with ID starting with c277fa9473f7d639e72279e0ab5dcc549999fb67b380b047b142b1d59ab11c8f not found: ID does not exist" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.923305 6275 scope.go:122] "RemoveContainer" containerID="d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.923606 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c"} err="failed to get container status \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": rpc error: code = NotFound desc = could not find container \"d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c\": container with ID starting with d67ea9d0a8d3520b595234d844be2890723c744e68c0733b4e564cdaaa47674c not found: ID does not exist" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.923637 6275 scope.go:122] "RemoveContainer" containerID="141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.923893 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85"} err="failed to get container status \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": rpc error: code = NotFound desc = could not find container \"141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85\": container with ID starting with 141e2a7dfd3a3b95573002f8c8f8342e480c4446188d93f6b1d52984205b5a85 not found: ID does not exist" Oct 30 01:29:40.924154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.923923 6275 scope.go:122] "RemoveContainer" containerID="e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70" Oct 30 01:29:40.924564 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.924304 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70"} err="failed to get container status \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": rpc error: code = NotFound desc = could not find container \"e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70\": container with ID starting with e96aee28f8efee9e7c32fc6ad06776e9352e014649a721e05573218d0c153c70 not found: ID does not exist" Oct 30 01:29:40.924564 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.924326 6275 scope.go:122] "RemoveContainer" containerID="7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.924775 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e"} err="failed to get container status \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": rpc error: code = NotFound desc = could not find container \"7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e\": container with ID starting with 7f487586fe181555ff600029b459220291f23a94928d340e0e6d87e9aab5ca4e not found: ID does not exist" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.924808 6275 scope.go:122] "RemoveContainer" containerID="7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.924969 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925483 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-web" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925501 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-web" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925526 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="prom-label-proxy" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925534 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="prom-label-proxy" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925549 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="config-reloader" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925556 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="config-reloader" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925574 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="alertmanager" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925581 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="alertmanager" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925612 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925620 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925641 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="init-config-reloader" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925649 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="init-config-reloader" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925665 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-metric" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925672 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-metric" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925755 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9"} err="failed to get container status \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": rpc error: code = NotFound desc = could not find container \"7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9\": container with ID starting with 7bdaff39e3feea7b17898bdf15fe1f5f70be60155e199cbb50f5832b5674a4d9 not found: ID does not exist" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925781 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="prom-label-proxy" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925793 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="config-reloader" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925803 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="alertmanager" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925818 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925828 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="init-config-reloader" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925842 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-metric" Oct 30 01:29:40.928618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.925852 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81463a-be01-4475-94b8-2a84429273b9" containerName="kube-rbac-proxy-web" Oct 30 01:29:40.953461 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.953439 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:40.957056 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.957039 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:40.980695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:40.980673 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 01:29:41.026415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026332 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 01:29:41.026415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026341 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 01:29:41.026415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026341 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 01:29:41.026415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026405 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 01:29:41.026415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026413 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 01:29:41.026695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026429 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 01:29:41.026695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026435 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 01:29:41.026695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026491 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bfmqc\"" Oct 30 01:29:41.026695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.026524 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 01:29:41.124169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124059 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124135 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124271 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124313 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124379 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3733d077-f21f-48bf-97f4-a3bca9cae0d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124414 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-web-config\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124436 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3733d077-f21f-48bf-97f4-a3bca9cae0d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124455 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3733d077-f21f-48bf-97f4-a3bca9cae0d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124473 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3733d077-f21f-48bf-97f4-a3bca9cae0d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124495 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124513 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3733d077-f21f-48bf-97f4-a3bca9cae0d8-config-out\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124527 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.124618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.124547 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vj8\" (UniqueName: \"kubernetes.io/projected/3733d077-f21f-48bf-97f4-a3bca9cae0d8-kube-api-access-d5vj8\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225713 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225671 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3733d077-f21f-48bf-97f4-a3bca9cae0d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225713 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225717 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225738 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3733d077-f21f-48bf-97f4-a3bca9cae0d8-config-out\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225762 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225784 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vj8\" (UniqueName: \"kubernetes.io/projected/3733d077-f21f-48bf-97f4-a3bca9cae0d8-kube-api-access-d5vj8\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225820 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.225974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.225847 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226247 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226333 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226289 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226329 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3733d077-f21f-48bf-97f4-a3bca9cae0d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226370 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-web-config\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226399 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3733d077-f21f-48bf-97f4-a3bca9cae0d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226426 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3733d077-f21f-48bf-97f4-a3bca9cae0d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226586 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226557 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3733d077-f21f-48bf-97f4-a3bca9cae0d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.226941 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.226845 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3733d077-f21f-48bf-97f4-a3bca9cae0d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.227634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.227605 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3733d077-f21f-48bf-97f4-a3bca9cae0d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.229398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.229012 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.229398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.229274 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.229398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.229370 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.229853 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.229827 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.230261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.230242 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3733d077-f21f-48bf-97f4-a3bca9cae0d8-config-out\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.230463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.230418 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.231370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.231348 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3733d077-f21f-48bf-97f4-a3bca9cae0d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.231800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.231784 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.232558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.232533 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3733d077-f21f-48bf-97f4-a3bca9cae0d8-web-config\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.248209 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.248142 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vj8\" (UniqueName: \"kubernetes.io/projected/3733d077-f21f-48bf-97f4-a3bca9cae0d8-kube-api-access-d5vj8\") pod \"alertmanager-main-0\" (UID: \"3733d077-f21f-48bf-97f4-a3bca9cae0d8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.331419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.331384 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:29:41.331579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.331484 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:29:41.331731 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.331717 6275 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:29:41.632648 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.632614 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:41.699940 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.699671 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:29:41.704296 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:41.704258 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3733d077_f21f_48bf_97f4_a3bca9cae0d8.slice/crio-7cabbfccb33ec126d34b8816fd4648bbe662e8aca75ae8e45855bd84c2e9c362 WatchSource:0}: Error finding container 7cabbfccb33ec126d34b8816fd4648bbe662e8aca75ae8e45855bd84c2e9c362: Status 404 returned error can't find the container with id 7cabbfccb33ec126d34b8816fd4648bbe662e8aca75ae8e45855bd84c2e9c362 Oct 30 01:29:41.735787 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.735761 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e81463a-be01-4475-94b8-2a84429273b9" path="/var/lib/kubelet/pods/6e81463a-be01-4475-94b8-2a84429273b9/volumes" Oct 30 01:29:41.843075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.842902 6275 generic.go:356] "Generic (PLEG): container finished" podID="3733d077-f21f-48bf-97f4-a3bca9cae0d8" containerID="dac7ec255491870b58514651f3b5829aa960713eed2855b54598a59b677c490e" exitCode=0 Oct 30 01:29:41.843075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.842954 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerDied","Data":"dac7ec255491870b58514651f3b5829aa960713eed2855b54598a59b677c490e"} Oct 30 01:29:41.843075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:41.843032 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"7cabbfccb33ec126d34b8816fd4648bbe662e8aca75ae8e45855bd84c2e9c362"} Oct 30 01:29:42.851376 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.851343 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"699ac3a50c922d7ef015cef5927b050b0fd9fbbba6d8395a249cbf5662006433"} Oct 30 01:29:42.851857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.851456 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"3786ab8394790ff827ac269ab45cc40af218df1bb7e19b77320bb42ef1481abd"} Oct 30 01:29:42.851857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.851495 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"e437b19894571ba2c08e68bcc48b41d0b724e227abee740c429a10f4c2ca959c"} Oct 30 01:29:42.851857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.851521 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"5aedec41a86cc52b447aba011635e7ae0647390949ecd7e1a1a9e3ebfeed49f7"} Oct 30 01:29:42.851857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.851551 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"acc784905107a74c8e847cf43b6c3b564967880a6596ca7351efd707593529a5"} Oct 30 01:29:42.851857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.851597 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3733d077-f21f-48bf-97f4-a3bca9cae0d8","Type":"ContainerStarted","Data":"2ed09b0bd66a7163554dddcfcd59519848008fe62ff447df7d5c0b06c75321d6"} Oct 30 01:29:42.903630 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:42.903579 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.890692686 podStartE2EDuration="2.903564795s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.872109ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:42.902957446 +0000 UTC m=+11.294765078" watchObservedRunningTime="2026-10-30 01:29:42.903564795 +0000 UTC m=+11.295372420" Oct 30 01:29:44.227904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.227862 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz"] Oct 30 01:29:44.248426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.248393 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.273494 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.273467 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-75xlc\"" Oct 30 01:29:44.273628 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.273561 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 01:29:44.273684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.273631 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 01:29:44.273684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.273667 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 01:29:44.273798 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.273702 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 01:29:44.274342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.274274 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz"] Oct 30 01:29:44.311744 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.311720 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 01:29:44.323016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.322995 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 01:29:44.354805 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.354774 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.354951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.354832 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-serving-certs-ca-bundle\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.354951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.354905 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6thql\" (UniqueName: \"kubernetes.io/projected/38137e76-29e8-4de7-b7fc-1367a7b551a3-kube-api-access-6thql\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.354951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.354936 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-federate-client-tls\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.355092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.354956 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.355092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.354985 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-metrics-client-ca\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.355092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.355013 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-telemeter-client-tls\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.355092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.355059 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-secret-telemeter-client\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.455998 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-serving-certs-ca-bundle\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456036 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6thql\" (UniqueName: \"kubernetes.io/projected/38137e76-29e8-4de7-b7fc-1367a7b551a3-kube-api-access-6thql\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456068 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-federate-client-tls\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456099 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456149 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-metrics-client-ca\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456169 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-telemeter-client-tls\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456215 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-secret-telemeter-client\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456254 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.456883 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.456823 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-serving-certs-ca-bundle\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.457030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.457002 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-metrics-client-ca\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.457220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.457194 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38137e76-29e8-4de7-b7fc-1367a7b551a3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.458976 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.458928 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.459163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.459116 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-federate-client-tls\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.459444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.459423 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-secret-telemeter-client\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.459526 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.459423 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/38137e76-29e8-4de7-b7fc-1367a7b551a3-telemeter-client-tls\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.478102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.478041 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6thql\" (UniqueName: \"kubernetes.io/projected/38137e76-29e8-4de7-b7fc-1367a7b551a3-kube-api-access-6thql\") pod \"telemeter-client-74bf55dd7c-nd7rz\" (UID: \"38137e76-29e8-4de7-b7fc-1367a7b551a3\") " pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.603245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.603214 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" Oct 30 01:29:44.970519 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:44.970476 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz"] Oct 30 01:29:44.975074 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:44.975047 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38137e76_29e8_4de7_b7fc_1367a7b551a3.slice/crio-9aaa7bf09197d30f9a57345436c1fe8bcd9262ab69363e7e549619203660f8fe WatchSource:0}: Error finding container 9aaa7bf09197d30f9a57345436c1fe8bcd9262ab69363e7e549619203660f8fe: Status 404 returned error can't find the container with id 9aaa7bf09197d30f9a57345436c1fe8bcd9262ab69363e7e549619203660f8fe Oct 30 01:29:45.771994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:45.771958 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:45.778097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:45.778069 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:45.865041 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:45.865002 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" event={"ID":"38137e76-29e8-4de7-b7fc-1367a7b551a3","Type":"ContainerStarted","Data":"9aaa7bf09197d30f9a57345436c1fe8bcd9262ab69363e7e549619203660f8fe"} Oct 30 01:29:45.869750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:45.869726 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:29:45.946046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:45.946016 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-545767f87d-ccnvd"] Oct 30 01:29:46.698834 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:29:46.698800 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:29:47.875691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:47.875653 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" event={"ID":"38137e76-29e8-4de7-b7fc-1367a7b551a3","Type":"ContainerStarted","Data":"421e8d498edf1894ee8bd41c931e0b6a22fa083580846a22b5464d1622553937"} Oct 30 01:29:47.876244 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:47.875714 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" event={"ID":"38137e76-29e8-4de7-b7fc-1367a7b551a3","Type":"ContainerStarted","Data":"805a3109fa76cd192f8b09461ea40c6def4db4aa63d3d11d3fe288a51c6d8db6"} Oct 30 01:29:47.876244 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:47.875734 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" event={"ID":"38137e76-29e8-4de7-b7fc-1367a7b551a3","Type":"ContainerStarted","Data":"ccb1b275716dce1b9ef2bef7c1ee086ba771da27acfadc903dd5c93ba0a75171"} Oct 30 01:29:47.961717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:47.961653 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74bf55dd7c-nd7rz" podStartSLOduration=1.854920133 podStartE2EDuration="3.961640236s" totalImagesPullingTime="2.106720103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:47.96075407 +0000 UTC m=+16.352561703" watchObservedRunningTime="2026-10-30 01:29:47.961640236 +0000 UTC m=+16.353447865" Oct 30 01:29:48.881955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:48.881913 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nc6vm" event={"ID":"ec32c8f1-eb5a-4a91-af80-f41d281d672e","Type":"ContainerStarted","Data":"fd58995af034f1f26fe04464710f10532bff70ca2560ae126532a496daf68b8b"} Oct 30 01:29:48.882641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:48.882614 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:48.884051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:48.884022 6275 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-nc6vm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body= Oct 30 01:29:48.884194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:48.884075 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-nc6vm" podUID="ec32c8f1-eb5a-4a91-af80-f41d281d672e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" Oct 30 01:29:49.902766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:49.902736 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nc6vm" Oct 30 01:29:50.069688 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.069654 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5497b9c4fb-tsbnw"] Oct 30 01:29:50.072797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.072772 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.095921 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.095887 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5497b9c4fb-tsbnw"] Oct 30 01:29:50.107909 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.107879 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw68t\" (UniqueName: \"kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108089 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.107933 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108089 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.108047 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108089 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.108066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.108143 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.108168 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.108191 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.108261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.108233 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.208874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.208783 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.208874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.208850 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.208874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.208872 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.208917 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.208945 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw68t\" (UniqueName: \"kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.209146 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.209224 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.209251 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209890 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.209859 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.209890 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.209882 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.210068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.210044 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.210430 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.210407 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.210599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.210581 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.213323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.213300 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.213412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.213344 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.231069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.231041 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw68t\" (UniqueName: \"kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t\") pod \"console-5497b9c4fb-tsbnw\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.385484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.385444 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:29:50.693450 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.693313 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5497b9c4fb-tsbnw"] Oct 30 01:29:50.698554 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:29:50.698524 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f47423b_1519_4521_a6bc_0f660ca5497a.slice/crio-3369ce8793b871e625471e4294e23e43d3f31e61ee0633cd0b479f924e03ff95 WatchSource:0}: Error finding container 3369ce8793b871e625471e4294e23e43d3f31e61ee0633cd0b479f924e03ff95: Status 404 returned error can't find the container with id 3369ce8793b871e625471e4294e23e43d3f31e61ee0633cd0b479f924e03ff95 Oct 30 01:29:50.892797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.892746 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5497b9c4fb-tsbnw" event={"ID":"9f47423b-1519-4521-a6bc-0f660ca5497a","Type":"ContainerStarted","Data":"d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69"} Oct 30 01:29:50.892986 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.892812 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5497b9c4fb-tsbnw" event={"ID":"9f47423b-1519-4521-a6bc-0f660ca5497a","Type":"ContainerStarted","Data":"3369ce8793b871e625471e4294e23e43d3f31e61ee0633cd0b479f924e03ff95"} Oct 30 01:29:50.933576 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:50.933519 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5497b9c4fb-tsbnw" podStartSLOduration=0.933499218 podStartE2EDuration="933.499218ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:50.932908784 +0000 UTC m=+19.324716422" watchObservedRunningTime="2026-10-30 01:29:50.933499218 +0000 UTC m=+19.325306853" Oct 30 01:29:52.983473 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:29:52.983168 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kd25" Oct 30 01:30:00.385609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:00.385570 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:30:00.386228 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:00.385663 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:30:00.390259 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:00.390235 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:30:00.934768 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:00.934741 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:30:01.010108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:01.010075 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7d4bbb89-6hl5t"] Oct 30 01:30:02.742082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:02.742050 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lwf9z" Oct 30 01:30:05.108447 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:05.108412 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9qlzc" Oct 30 01:30:10.967062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:10.967003 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-545767f87d-ccnvd" podUID="9b65ce43-3ae8-461b-b703-fbc0b37645ea" containerName="console" containerID="cri-o://ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3" gracePeriod=15 Oct 30 01:30:11.230024 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.229948 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:30:11.286577 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286549 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.286767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286590 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.286767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286638 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.286767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286681 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.286767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286713 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.286969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286788 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.286969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.286845 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") pod \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\" (UID: \"9b65ce43-3ae8-461b-b703-fbc0b37645ea\") " Oct 30 01:30:11.287271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.287234 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:11.287271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.287250 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.287271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.287258 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.287445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.287310 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.289098 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.289069 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:11.289229 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.289211 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:11.289273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.289250 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87" pod "9b65ce43-3ae8-461b-b703-fbc0b37645ea" (UID: "9b65ce43-3ae8-461b-b703-fbc0b37645ea"). InnerVolumeSpecName "kube-api-access-dkx87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.388399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388366 6275 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-oauth-serving-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388396 6275 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388405 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkx87\" (UniqueName: \"kubernetes.io/projected/9b65ce43-3ae8-461b-b703-fbc0b37645ea-kube-api-access-dkx87\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388414 6275 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b65ce43-3ae8-461b-b703-fbc0b37645ea-tmp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388424 6275 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-serving-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388433 6275 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b65ce43-3ae8-461b-b703-fbc0b37645ea-console-oauth-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.388441 6275 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b65ce43-3ae8-461b-b703-fbc0b37645ea-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.437478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.437448 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d7f46558-w6nbx"] Oct 30 01:30:11.437731 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.437718 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9b65ce43-3ae8-461b-b703-fbc0b37645ea" containerName="console" Oct 30 01:30:11.437777 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.437732 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b65ce43-3ae8-461b-b703-fbc0b37645ea" containerName="console" Oct 30 01:30:11.437819 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.437794 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b65ce43-3ae8-461b-b703-fbc0b37645ea" containerName="console" Oct 30 01:30:11.461777 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.461751 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.465101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.465080 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d7f46558-w6nbx"] Oct 30 01:30:11.590787 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.590757 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-service-ca\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.590956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.590804 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflj6\" (UniqueName: \"kubernetes.io/projected/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-kube-api-access-nflj6\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.590956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.590893 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-config\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.590956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.590931 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-tmp\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.591062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.590957 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-serving-cert\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.591062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.590980 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-oauth-config\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.591062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.591051 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-trusted-ca-bundle\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.591182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.591066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-oauth-serving-cert\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692314 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-config\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692364 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-tmp\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692554 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692481 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-serving-cert\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692554 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692519 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-oauth-config\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692557 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-trusted-ca-bundle\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692581 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-oauth-serving-cert\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692632 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-service-ca\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.692790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.692693 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflj6\" (UniqueName: \"kubernetes.io/projected/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-kube-api-access-nflj6\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.693073 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.693036 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-tmp\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.693226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.693179 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-config\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.693385 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.693367 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-service-ca\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.693443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.693420 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-oauth-serving-cert\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.693566 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.693541 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-trusted-ca-bundle\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.695051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.695030 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-serving-cert\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.695051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.695046 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-console-oauth-config\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.714493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.714469 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflj6\" (UniqueName: \"kubernetes.io/projected/070fcbc8-7d63-4a37-b073-f0b8ed68f2db-kube-api-access-nflj6\") pod \"console-77d7f46558-w6nbx\" (UID: \"070fcbc8-7d63-4a37-b073-f0b8ed68f2db\") " pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.784458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.784429 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:11.967361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.967325 6275 generic.go:356] "Generic (PLEG): container finished" podID="9b65ce43-3ae8-461b-b703-fbc0b37645ea" containerID="ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3" exitCode=0 Oct 30 01:30:11.967850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.967382 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545767f87d-ccnvd" event={"ID":"9b65ce43-3ae8-461b-b703-fbc0b37645ea","Type":"ContainerDied","Data":"ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3"} Oct 30 01:30:11.967850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.967390 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545767f87d-ccnvd" Oct 30 01:30:11.967850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.967425 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545767f87d-ccnvd" event={"ID":"9b65ce43-3ae8-461b-b703-fbc0b37645ea","Type":"ContainerDied","Data":"ee8eb3c35f76cdc9494aef7d9e6487a6f855434b2cdb9ad6ef9b9a18e522920c"} Oct 30 01:30:11.967850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.967445 6275 scope.go:122] "RemoveContainer" containerID="ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3" Oct 30 01:30:11.975000 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.974978 6275 scope.go:122] "RemoveContainer" containerID="ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3" Oct 30 01:30:11.975289 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:30:11.975268 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3\": container with ID starting with ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3 not found: ID does not exist" containerID="ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3" Oct 30 01:30:11.975357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:11.975301 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3"} err="failed to get container status \"ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3\": rpc error: code = NotFound desc = could not find container \"ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3\": container with ID starting with ba4a646f9be44208555e4671023b79fece8c9b808983ca3445b47ebceb214ba3 not found: ID does not exist" Oct 30 01:30:12.012929 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:12.012897 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-545767f87d-ccnvd"] Oct 30 01:30:12.018823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:12.018796 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-545767f87d-ccnvd"] Oct 30 01:30:12.102728 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:12.102694 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d7f46558-w6nbx"] Oct 30 01:30:12.107791 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:30:12.107763 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070fcbc8_7d63_4a37_b073_f0b8ed68f2db.slice/crio-80f7065e055c0fdbf50fb0de1e3c632e9ab0f9495a4a7b5fda01556ee5d5f5a5 WatchSource:0}: Error finding container 80f7065e055c0fdbf50fb0de1e3c632e9ab0f9495a4a7b5fda01556ee5d5f5a5: Status 404 returned error can't find the container with id 80f7065e055c0fdbf50fb0de1e3c632e9ab0f9495a4a7b5fda01556ee5d5f5a5 Oct 30 01:30:12.972421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:12.972388 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d7f46558-w6nbx" event={"ID":"070fcbc8-7d63-4a37-b073-f0b8ed68f2db","Type":"ContainerStarted","Data":"17190ed82818122fc8f00e3df408e4780e0811a5186cc0bd741086b49490072a"} Oct 30 01:30:12.972911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:12.972437 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d7f46558-w6nbx" event={"ID":"070fcbc8-7d63-4a37-b073-f0b8ed68f2db","Type":"ContainerStarted","Data":"80f7065e055c0fdbf50fb0de1e3c632e9ab0f9495a4a7b5fda01556ee5d5f5a5"} Oct 30 01:30:13.019092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:13.019046 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77d7f46558-w6nbx" podStartSLOduration=2.019033505 podStartE2EDuration="2.019033505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:13.018973039 +0000 UTC m=+41.410780669" watchObservedRunningTime="2026-10-30 01:30:13.019033505 +0000 UTC m=+41.410841137" Oct 30 01:30:13.736138 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:13.736094 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b65ce43-3ae8-461b-b703-fbc0b37645ea" path="/var/lib/kubelet/pods/9b65ce43-3ae8-461b-b703-fbc0b37645ea/volumes" Oct 30 01:30:16.694794 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:30:16.694757 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:30:21.784944 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:21.784907 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:21.785359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:21.784968 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:21.789366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:21.789345 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:22.011274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:22.011249 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d7f46558-w6nbx" Oct 30 01:30:22.087303 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:22.087220 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5497b9c4fb-tsbnw"] Oct 30 01:30:26.029758 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.029716 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d7d4bbb89-6hl5t" podUID="ef9c23ee-3b0a-406e-8f9b-46b56e98854b" containerName="console" containerID="cri-o://6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f" gracePeriod=15 Oct 30 01:30:26.291931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.291842 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:30:26.419438 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419407 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419458 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419498 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419529 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419592 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419782 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419619 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6\" (UniqueName: \"kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419782 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419685 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419782 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419710 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config\") pod \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\" (UID: \"ef9c23ee-3b0a-406e-8f9b-46b56e98854b\") " Oct 30 01:30:26.419972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419771 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.419972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419966 6275 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-tmp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.420099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.419966 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.420189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.420163 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.420252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.420179 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.420252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.420203 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.421681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.421657 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:26.421802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.421717 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "kube-api-access-cn7l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:26.421802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.421763 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config" pod "ef9c23ee-3b0a-406e-8f9b-46b56e98854b" (UID: "ef9c23ee-3b0a-406e-8f9b-46b56e98854b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:26.521339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521299 6275 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-serving-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.521339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521333 6275 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-oauth-serving-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.521339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521346 6275 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-trusted-ca-bundle\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.521586 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521360 6275 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.521586 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521372 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn7l6\" (UniqueName: \"kubernetes.io/projected/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-kube-api-access-cn7l6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.521586 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521401 6275 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-oauth-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.521586 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:26.521425 6275 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef9c23ee-3b0a-406e-8f9b-46b56e98854b-console-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:27.025612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.025577 6275 generic.go:356] "Generic (PLEG): container finished" podID="ef9c23ee-3b0a-406e-8f9b-46b56e98854b" containerID="6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f" exitCode=0 Oct 30 01:30:27.025790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.025633 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7d4bbb89-6hl5t" Oct 30 01:30:27.025790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.025640 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d4bbb89-6hl5t" event={"ID":"ef9c23ee-3b0a-406e-8f9b-46b56e98854b","Type":"ContainerDied","Data":"6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f"} Oct 30 01:30:27.025790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.025701 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7d4bbb89-6hl5t" event={"ID":"ef9c23ee-3b0a-406e-8f9b-46b56e98854b","Type":"ContainerDied","Data":"2d4e02586d71a45740c8760d35197db0d3b9d26facb9197da1e50aa67ab2d672"} Oct 30 01:30:27.025790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.025720 6275 scope.go:122] "RemoveContainer" containerID="6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f" Oct 30 01:30:27.033726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.033592 6275 scope.go:122] "RemoveContainer" containerID="6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f" Oct 30 01:30:27.033946 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:30:27.033857 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f\": container with ID starting with 6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f not found: ID does not exist" containerID="6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f" Oct 30 01:30:27.033946 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.033885 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f"} err="failed to get container status \"6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f\": rpc error: code = NotFound desc = could not find container \"6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f\": container with ID starting with 6e9e693e13db9eb9e4f284c088a4a3d282a58dccd9546b3978cdfe242681b21f not found: ID does not exist" Oct 30 01:30:27.071893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.071868 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7d4bbb89-6hl5t"] Oct 30 01:30:27.082328 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.082303 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d7d4bbb89-6hl5t"] Oct 30 01:30:27.735992 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:27.735956 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9c23ee-3b0a-406e-8f9b-46b56e98854b" path="/var/lib/kubelet/pods/ef9c23ee-3b0a-406e-8f9b-46b56e98854b/volumes" Oct 30 01:30:46.694404 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:30:46.694363 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:30:47.108160 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.108092 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5497b9c4fb-tsbnw" podUID="9f47423b-1519-4521-a6bc-0f660ca5497a" containerName="console" containerID="cri-o://d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69" gracePeriod=15 Oct 30 01:30:47.361974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.361914 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:30:47.376717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376688 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.376717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376716 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.376912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376742 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.376912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376770 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t\" (UniqueName: \"kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.376912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376891 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.377068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376944 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.377068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.376968 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.377068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377040 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert\") pod \"9f47423b-1519-4521-a6bc-0f660ca5497a\" (UID: \"9f47423b-1519-4521-a6bc-0f660ca5497a\") " Oct 30 01:30:47.377247 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377078 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.377247 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377095 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:47.377431 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377396 6275 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-trusted-ca-bundle\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.377431 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377422 6275 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47423b-1519-4521-a6bc-0f660ca5497a-tmp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.377553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377411 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:47.377553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377430 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:47.377553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.377420 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:47.378978 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.378955 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:47.379071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.378978 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:47.379071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.379035 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t" pod "9f47423b-1519-4521-a6bc-0f660ca5497a" (UID: "9f47423b-1519-4521-a6bc-0f660ca5497a"). InnerVolumeSpecName "kube-api-access-jw68t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:47.478227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.478187 6275 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-oauth-serving-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.478227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.478221 6275 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-oauth-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.478227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.478232 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw68t\" (UniqueName: \"kubernetes.io/projected/9f47423b-1519-4521-a6bc-0f660ca5497a-kube-api-access-jw68t\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.478475 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.478241 6275 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f47423b-1519-4521-a6bc-0f660ca5497a-console-serving-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.478475 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.478253 6275 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-console-config\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.478475 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:47.478262 6275 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f47423b-1519-4521-a6bc-0f660ca5497a-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:30:48.100465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.100431 6275 generic.go:356] "Generic (PLEG): container finished" podID="9f47423b-1519-4521-a6bc-0f660ca5497a" containerID="d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69" exitCode=0 Oct 30 01:30:48.100938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.100487 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5497b9c4fb-tsbnw" Oct 30 01:30:48.100938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.100521 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5497b9c4fb-tsbnw" event={"ID":"9f47423b-1519-4521-a6bc-0f660ca5497a","Type":"ContainerDied","Data":"d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69"} Oct 30 01:30:48.100938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.100588 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5497b9c4fb-tsbnw" event={"ID":"9f47423b-1519-4521-a6bc-0f660ca5497a","Type":"ContainerDied","Data":"3369ce8793b871e625471e4294e23e43d3f31e61ee0633cd0b479f924e03ff95"} Oct 30 01:30:48.100938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.100622 6275 scope.go:122] "RemoveContainer" containerID="d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69" Oct 30 01:30:48.108256 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.108241 6275 scope.go:122] "RemoveContainer" containerID="d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69" Oct 30 01:30:48.108515 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:30:48.108497 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69\": container with ID starting with d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69 not found: ID does not exist" containerID="d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69" Oct 30 01:30:48.108556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.108524 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69"} err="failed to get container status \"d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69\": rpc error: code = NotFound desc = could not find container \"d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69\": container with ID starting with d5a7dd48810df0c1d73cf73756e415b2cb8a2f56c1dfc34879beee5799cf7a69 not found: ID does not exist" Oct 30 01:30:48.145477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.145443 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5497b9c4fb-tsbnw"] Oct 30 01:30:48.156129 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:48.156103 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5497b9c4fb-tsbnw"] Oct 30 01:30:49.735890 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:30:49.735858 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f47423b-1519-4521-a6bc-0f660ca5497a" path="/var/lib/kubelet/pods/9f47423b-1519-4521-a6bc-0f660ca5497a/volumes" Oct 30 01:31:16.694709 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:31:16.694624 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:31:46.694971 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:31:46.694929 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:32:16.695315 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:32:16.695276 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:32:46.694719 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:32:46.694641 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:32:47.029432 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029399 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl"] Oct 30 01:32:47.029742 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029726 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ef9c23ee-3b0a-406e-8f9b-46b56e98854b" containerName="console" Oct 30 01:32:47.029828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029743 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9c23ee-3b0a-406e-8f9b-46b56e98854b" containerName="console" Oct 30 01:32:47.029828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029774 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9f47423b-1519-4521-a6bc-0f660ca5497a" containerName="console" Oct 30 01:32:47.029828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029780 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f47423b-1519-4521-a6bc-0f660ca5497a" containerName="console" Oct 30 01:32:47.029828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029845 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9c23ee-3b0a-406e-8f9b-46b56e98854b" containerName="console" Oct 30 01:32:47.029828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.029856 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f47423b-1519-4521-a6bc-0f660ca5497a" containerName="console" Oct 30 01:32:47.031827 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.031801 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.057283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.057256 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 01:32:47.057283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.057265 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 01:32:47.057461 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.057334 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wl2r9\"" Oct 30 01:32:47.084686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.084658 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl"] Oct 30 01:32:47.158809 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.158781 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.158993 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.158838 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.158993 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.158902 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bct\" (UniqueName: \"kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.259976 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.259944 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.260182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.259993 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.260182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.260024 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bct\" (UniqueName: \"kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.260453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.260433 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.260495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.260443 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.281768 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.281701 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bct\" (UniqueName: \"kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.344154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.344104 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:32:47.530715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.530679 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp"] Oct 30 01:32:47.538077 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.538015 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.557816 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.557787 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:32:47.557953 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.557845 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:32:47.557953 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.557873 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:32:47.558041 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.557977 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v6t8d\"" Oct 30 01:32:47.558330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.558317 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:32:47.558401 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.558346 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:32:47.558546 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.558533 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:32:47.571945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.571921 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp"] Oct 30 01:32:47.642435 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.642407 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl"] Oct 30 01:32:47.647778 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:32:47.647753 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2490016f_bf9e_4fb6_9fd9_3402b81d2453.slice/crio-3d45737ba8df6ad558086bb6f51c32a1a569b80c05acd8121e694ae1f4c491a6 WatchSource:0}: Error finding container 3d45737ba8df6ad558086bb6f51c32a1a569b80c05acd8121e694ae1f4c491a6: Status 404 returned error can't find the container with id 3d45737ba8df6ad558086bb6f51c32a1a569b80c05acd8121e694ae1f4c491a6 Oct 30 01:32:47.662627 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662605 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.662735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662643 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.662735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662660 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgc4\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.662735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662695 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.662875 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662777 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.662948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662924 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.663005 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.662991 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.763546 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763517 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763558 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763578 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgc4\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763597 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763619 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763658 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.763701 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.764449 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.764426 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.766024 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.766002 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.766156 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.766053 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.766302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.766283 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.766361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.766334 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.787489 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.787464 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.787680 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.787664 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgc4\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4\") pod \"istiod-openshift-gateway-7876bfcb74-2gzsp\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:47.848717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:47.848672 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:48.161415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:48.161326 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp"] Oct 30 01:32:48.165614 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:32:48.165587 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04b1405_2e5e_492c_8da4_94b0ae52267b.slice/crio-0ac138b11c2524c6a8a1391c676cbee05db219b300027275b3b80dafc8e92ff7 WatchSource:0}: Error finding container 0ac138b11c2524c6a8a1391c676cbee05db219b300027275b3b80dafc8e92ff7: Status 404 returned error can't find the container with id 0ac138b11c2524c6a8a1391c676cbee05db219b300027275b3b80dafc8e92ff7 Oct 30 01:32:48.508708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:48.508626 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" event={"ID":"2490016f-bf9e-4fb6-9fd9-3402b81d2453","Type":"ContainerStarted","Data":"3d45737ba8df6ad558086bb6f51c32a1a569b80c05acd8121e694ae1f4c491a6"} Oct 30 01:32:48.509955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:48.509925 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" event={"ID":"d04b1405-2e5e-492c-8da4-94b0ae52267b","Type":"ContainerStarted","Data":"0ac138b11c2524c6a8a1391c676cbee05db219b300027275b3b80dafc8e92ff7"} Oct 30 01:32:50.843993 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:50.843955 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:32:50.844280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:50.844024 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:32:51.523594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:51.523562 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" event={"ID":"d04b1405-2e5e-492c-8da4-94b0ae52267b","Type":"ContainerStarted","Data":"1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2"} Oct 30 01:32:51.523791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:51.523771 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:51.525407 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:51.525371 6275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2gzsp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:32:51.525534 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:51.525423 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:32:51.598496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:51.598438 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" podStartSLOduration=1.922859457 podStartE2EDuration="4.598425057s" totalImagesPullingTime="2.6755656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:51.591019871 +0000 UTC m=+199.982827504" watchObservedRunningTime="2026-10-30 01:32:51.598425057 +0000 UTC m=+199.990232684" Oct 30 01:32:52.527841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:52.527798 6275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2gzsp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:32:52.528266 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:52.527869 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:32:55.527789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:55.527751 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:32:55.539622 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:55.539589 6275 generic.go:356] "Generic (PLEG): container finished" podID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerID="7e2a76a898a74b6c6f224644694ff9af7154351da4ddfbce758aa3aafeb5e01a" exitCode=0 Oct 30 01:32:55.539894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:55.539675 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" event={"ID":"2490016f-bf9e-4fb6-9fd9-3402b81d2453","Type":"ContainerDied","Data":"7e2a76a898a74b6c6f224644694ff9af7154351da4ddfbce758aa3aafeb5e01a"} Oct 30 01:32:58.552705 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:58.552673 6275 generic.go:356] "Generic (PLEG): container finished" podID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerID="c38ea82aeb89aedeff9d865a773431e09810e8bcb190c23f410389729bfc09a9" exitCode=0 Oct 30 01:32:58.553173 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:32:58.552766 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" event={"ID":"2490016f-bf9e-4fb6-9fd9-3402b81d2453","Type":"ContainerDied","Data":"c38ea82aeb89aedeff9d865a773431e09810e8bcb190c23f410389729bfc09a9"} Oct 30 01:33:04.577746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:04.577714 6275 generic.go:356] "Generic (PLEG): container finished" podID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerID="62de97b0896de2ed6315f43b4bf1b17ba9e455974c3d6e6356ccf78c66a60426" exitCode=0 Oct 30 01:33:04.578175 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:04.577755 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" event={"ID":"2490016f-bf9e-4fb6-9fd9-3402b81d2453","Type":"ContainerDied","Data":"62de97b0896de2ed6315f43b4bf1b17ba9e455974c3d6e6356ccf78c66a60426"} Oct 30 01:33:05.707240 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.707216 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:33:05.829309 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.829277 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct\" (UniqueName: \"kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct\") pod \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " Oct 30 01:33:05.829484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.829353 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle\") pod \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " Oct 30 01:33:05.829484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.829382 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util\") pod \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\" (UID: \"2490016f-bf9e-4fb6-9fd9-3402b81d2453\") " Oct 30 01:33:05.830169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.830115 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle" pod "2490016f-bf9e-4fb6-9fd9-3402b81d2453" (UID: "2490016f-bf9e-4fb6-9fd9-3402b81d2453"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:05.831527 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.831499 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct" pod "2490016f-bf9e-4fb6-9fd9-3402b81d2453" (UID: "2490016f-bf9e-4fb6-9fd9-3402b81d2453"). InnerVolumeSpecName "kube-api-access-g4bct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:05.833773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.833730 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util" pod "2490016f-bf9e-4fb6-9fd9-3402b81d2453" (UID: "2490016f-bf9e-4fb6-9fd9-3402b81d2453"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:05.930505 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.930472 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bct\" (UniqueName: \"kubernetes.io/projected/2490016f-bf9e-4fb6-9fd9-3402b81d2453-kube-api-access-g4bct\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:33:05.930505 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.930501 6275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-bundle\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:33:05.930505 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:05.930510 6275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2490016f-bf9e-4fb6-9fd9-3402b81d2453-util\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:33:06.587206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:06.587169 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" event={"ID":"2490016f-bf9e-4fb6-9fd9-3402b81d2453","Type":"ContainerDied","Data":"3d45737ba8df6ad558086bb6f51c32a1a569b80c05acd8121e694ae1f4c491a6"} Oct 30 01:33:06.587386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:06.587229 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d45737ba8df6ad558086bb6f51c32a1a569b80c05acd8121e694ae1f4c491a6" Oct 30 01:33:06.587386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:06.587185 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bsbdl" Oct 30 01:33:16.694405 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:33:16.694371 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:33:29.946593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946555 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk"] Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946853 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="extract" Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946866 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="extract" Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946876 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="pull" Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946881 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="pull" Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946907 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="util" Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.946913 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="util" Oct 30 01:33:29.948226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.947000 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2490016f-bf9e-4fb6-9fd9-3402b81d2453" containerName="extract" Oct 30 01:33:29.949074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.949057 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:29.977808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.977783 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 01:33:29.977954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.977806 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 01:33:29.977954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.977874 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-79ppx\"" Oct 30 01:33:29.977954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.977879 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 01:33:29.979859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:29.979836 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk"] Oct 30 01:33:30.009754 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.009728 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.009910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.009760 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.009910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.009781 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.009910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.009799 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwm9c\" (UniqueName: \"kubernetes.io/projected/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-kube-api-access-qwm9c\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.010069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.009959 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.010069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.010019 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.044176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.044144 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 01:33:30.044363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.044303 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 01:33:30.110915 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.110883 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.110915 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.110916 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.111159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.110935 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.111159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.110956 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwm9c\" (UniqueName: \"kubernetes.io/projected/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-kube-api-access-qwm9c\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.111159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.110983 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.111159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.111007 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.111509 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.111479 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.113462 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.113435 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.113604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.113567 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.113696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.113675 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.113745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.113689 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.135352 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.135324 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwm9c\" (UniqueName: \"kubernetes.io/projected/017c1081-7b28-4ad6-932d-3ff9ae1d3ed6-kube-api-access-qwm9c\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-dw6pk\" (UID: \"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.262459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.262426 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:30.630086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.629882 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk"] Oct 30 01:33:30.633102 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:33:30.633071 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017c1081_7b28_4ad6_932d_3ff9ae1d3ed6.slice/crio-cd9bf3413ec9748a63bc6f22d2616bb365c0ec42001db81f64f43d3015a8118d WatchSource:0}: Error finding container cd9bf3413ec9748a63bc6f22d2616bb365c0ec42001db81f64f43d3015a8118d: Status 404 returned error can't find the container with id cd9bf3413ec9748a63bc6f22d2616bb365c0ec42001db81f64f43d3015a8118d Oct 30 01:33:30.669183 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:30.669153 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" event={"ID":"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6","Type":"ContainerStarted","Data":"cd9bf3413ec9748a63bc6f22d2616bb365c0ec42001db81f64f43d3015a8118d"} Oct 30 01:33:34.690038 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:34.690001 6275 generic.go:356] "Generic (PLEG): container finished" podID="017c1081-7b28-4ad6-932d-3ff9ae1d3ed6" containerID="077e3d38576bff8962c8e08b6447892831e6d85996d07453b18a38671e44e0ab" exitCode=0 Oct 30 01:33:34.690444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:34.690055 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" event={"ID":"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6","Type":"ContainerDied","Data":"077e3d38576bff8962c8e08b6447892831e6d85996d07453b18a38671e44e0ab"} Oct 30 01:33:35.696177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:35.696143 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" event={"ID":"017c1081-7b28-4ad6-932d-3ff9ae1d3ed6","Type":"ContainerStarted","Data":"e6348a11f13022fa02493447c30503bb013397db7f0887c983d60fb979cfe3e3"} Oct 30 01:33:35.696559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:35.696279 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:33:35.786521 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:35.786463 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" podStartSLOduration=3.036007924 podStartE2EDuration="6.786451074s" totalImagesPullingTime="3.294103516s" totalInitContainerRuntime="456.339634ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:35.775624913 +0000 UTC m=+244.167432547" watchObservedRunningTime="2026-10-30 01:33:35.786451074 +0000 UTC m=+244.178258707" Oct 30 01:33:46.695998 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:33:46.695959 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:33:46.702875 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:33:46.702853 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-dw6pk" Oct 30 01:34:01.510254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.510220 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8"] Oct 30 01:34:01.513165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.513141 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.540546 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.540517 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 01:34:01.540981 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.540954 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 01:34:01.550972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.550949 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ljz6n\"" Oct 30 01:34:01.552483 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.552464 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8"] Oct 30 01:34:01.590186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.590153 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807e1be9-1186-4cdc-a03b-7739c1720321-tls-certs\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.590349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.590266 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/807e1be9-1186-4cdc-a03b-7739c1720321-tmp\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.590349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.590322 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4s6n\" (UniqueName: \"kubernetes.io/projected/807e1be9-1186-4cdc-a03b-7739c1720321-kube-api-access-x4s6n\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.690986 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.690944 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/807e1be9-1186-4cdc-a03b-7739c1720321-tmp\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.691215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.690995 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4s6n\" (UniqueName: \"kubernetes.io/projected/807e1be9-1186-4cdc-a03b-7739c1720321-kube-api-access-x4s6n\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.691215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.691063 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807e1be9-1186-4cdc-a03b-7739c1720321-tls-certs\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.693247 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.693217 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/807e1be9-1186-4cdc-a03b-7739c1720321-tmp\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.693496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.693475 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807e1be9-1186-4cdc-a03b-7739c1720321-tls-certs\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.728778 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.728733 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4s6n\" (UniqueName: \"kubernetes.io/projected/807e1be9-1186-4cdc-a03b-7739c1720321-kube-api-access-x4s6n\") pod \"kube-auth-proxy-7bd6fc9ff9-m9zg8\" (UID: \"807e1be9-1186-4cdc-a03b-7739c1720321\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:01.825096 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:01.825023 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" Oct 30 01:34:02.122588 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:02.122501 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8"] Oct 30 01:34:02.127538 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:34:02.127509 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807e1be9_1186_4cdc_a03b_7739c1720321.slice/crio-d50d016e3a1b646f3427a910130ed0556ec712d0b714046771e182ff1756b310 WatchSource:0}: Error finding container d50d016e3a1b646f3427a910130ed0556ec712d0b714046771e182ff1756b310: Status 404 returned error can't find the container with id d50d016e3a1b646f3427a910130ed0556ec712d0b714046771e182ff1756b310 Oct 30 01:34:02.792051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:02.792011 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" event={"ID":"807e1be9-1186-4cdc-a03b-7739c1720321","Type":"ContainerStarted","Data":"d50d016e3a1b646f3427a910130ed0556ec712d0b714046771e182ff1756b310"} Oct 30 01:34:04.293027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.292990 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-b27jr"] Oct 30 01:34:04.384294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.384267 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b27jr"] Oct 30 01:34:04.384444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.384362 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.400575 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.400552 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 01:34:04.410975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.410953 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 01:34:04.411202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.411184 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-h2qxl\"" Oct 30 01:34:04.513213 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.513171 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zkk\" (UniqueName: \"kubernetes.io/projected/a199fc5a-b796-457b-8e56-d61bae6cc3dd-kube-api-access-87zkk\") pod \"cert-manager-597b95cb84-b27jr\" (UID: \"a199fc5a-b796-457b-8e56-d61bae6cc3dd\") " pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.513402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.513254 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a199fc5a-b796-457b-8e56-d61bae6cc3dd-bound-sa-token\") pod \"cert-manager-597b95cb84-b27jr\" (UID: \"a199fc5a-b796-457b-8e56-d61bae6cc3dd\") " pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.614585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.614491 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zkk\" (UniqueName: \"kubernetes.io/projected/a199fc5a-b796-457b-8e56-d61bae6cc3dd-kube-api-access-87zkk\") pod \"cert-manager-597b95cb84-b27jr\" (UID: \"a199fc5a-b796-457b-8e56-d61bae6cc3dd\") " pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.614756 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.614588 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a199fc5a-b796-457b-8e56-d61bae6cc3dd-bound-sa-token\") pod \"cert-manager-597b95cb84-b27jr\" (UID: \"a199fc5a-b796-457b-8e56-d61bae6cc3dd\") " pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.646611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.646582 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a199fc5a-b796-457b-8e56-d61bae6cc3dd-bound-sa-token\") pod \"cert-manager-597b95cb84-b27jr\" (UID: \"a199fc5a-b796-457b-8e56-d61bae6cc3dd\") " pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.646837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.646817 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zkk\" (UniqueName: \"kubernetes.io/projected/a199fc5a-b796-457b-8e56-d61bae6cc3dd-kube-api-access-87zkk\") pod \"cert-manager-597b95cb84-b27jr\" (UID: \"a199fc5a-b796-457b-8e56-d61bae6cc3dd\") " pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:04.719989 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:04.719950 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b27jr" Oct 30 01:34:05.020284 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:05.020250 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b27jr"] Oct 30 01:34:05.024966 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:34:05.024936 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda199fc5a_b796_457b_8e56_d61bae6cc3dd.slice/crio-9cd972ef4f0aca4e806219fdae447de59cdd422773a4ac9494529413ee1b134f WatchSource:0}: Error finding container 9cd972ef4f0aca4e806219fdae447de59cdd422773a4ac9494529413ee1b134f: Status 404 returned error can't find the container with id 9cd972ef4f0aca4e806219fdae447de59cdd422773a4ac9494529413ee1b134f Oct 30 01:34:05.807227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:05.807194 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" event={"ID":"807e1be9-1186-4cdc-a03b-7739c1720321","Type":"ContainerStarted","Data":"fb25a52b9b9e2c457e8c9bf2a8445b2977e0c4b7b3c5195149e30736ab1458e1"} Oct 30 01:34:05.808677 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:05.808620 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b27jr" event={"ID":"a199fc5a-b796-457b-8e56-d61bae6cc3dd","Type":"ContainerStarted","Data":"9cd972ef4f0aca4e806219fdae447de59cdd422773a4ac9494529413ee1b134f"} Oct 30 01:34:05.846102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:05.846045 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-m9zg8" podStartSLOduration=2.178689239 podStartE2EDuration="4.846029415s" totalImagesPullingTime="2.667340176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:05.845556656 +0000 UTC m=+274.237364294" watchObservedRunningTime="2026-10-30 01:34:05.846029415 +0000 UTC m=+274.237837047" Oct 30 01:34:08.822580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:08.822531 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b27jr" event={"ID":"a199fc5a-b796-457b-8e56-d61bae6cc3dd","Type":"ContainerStarted","Data":"f895a44b5ae5521bbec3d155f7c773c9d428e1331de1852f262dbf7451e98aef"} Oct 30 01:34:08.859446 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:08.859399 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-b27jr" podStartSLOduration=2.051414142 podStartE2EDuration="4.859385805s" totalImagesPullingTime="2.807971663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:08.85913229 +0000 UTC m=+277.250939915" watchObservedRunningTime="2026-10-30 01:34:08.859385805 +0000 UTC m=+277.251193415" Oct 30 01:34:16.695946 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:34:16.695858 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:34:31.746184 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:31.746157 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:34:31.746722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:34:31.746702 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:34:46.695253 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:34:46.695215 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:35:05.187945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.187909 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt"] Oct 30 01:35:05.190426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.190405 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.223665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.223619 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt"] Oct 30 01:35:05.317555 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317513 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.317734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317567 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.317734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317663 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.317734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317693 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.317734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317720 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.317938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317758 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.317938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.317839 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcljr\" (UniqueName: \"kubernetes.io/projected/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-kube-api-access-dcljr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.418820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418783 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.418820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418824 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.419060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418862 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.419060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418882 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcljr\" (UniqueName: \"kubernetes.io/projected/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-kube-api-access-dcljr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.419060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418923 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.419060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418953 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.419060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.418978 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.420081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.420051 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.421314 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.421286 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.421399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.421324 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.421399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.421379 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.421465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.421441 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.440426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.440364 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.440629 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.440610 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcljr\" (UniqueName: \"kubernetes.io/projected/e7ec1529-3bf9-4ded-8a38-77e9b1b7964d-kube-api-access-dcljr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5vjwt\" (UID: \"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.501613 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.501579 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:05.827541 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.827504 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt"] Oct 30 01:35:05.831932 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:35:05.831898 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ec1529_3bf9_4ded_8a38_77e9b1b7964d.slice/crio-d99b0ee4da49ac94c055a75943a48454315cc441eebe471f33882b403b131aa1 WatchSource:0}: Error finding container d99b0ee4da49ac94c055a75943a48454315cc441eebe471f33882b403b131aa1: Status 404 returned error can't find the container with id d99b0ee4da49ac94c055a75943a48454315cc441eebe471f33882b403b131aa1 Oct 30 01:35:05.833931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.833897 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:35:05.834022 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:05.833972 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:35:06.039419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:06.039386 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" event={"ID":"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d","Type":"ContainerStarted","Data":"5b74701e3b4aec088eea6ee9a38fbf5cfddb450eb4a81c7b6dcd19beb81c6b30"} Oct 30 01:35:06.039603 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:06.039442 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" event={"ID":"e7ec1529-3bf9-4ded-8a38-77e9b1b7964d","Type":"ContainerStarted","Data":"d99b0ee4da49ac94c055a75943a48454315cc441eebe471f33882b403b131aa1"} Oct 30 01:35:06.039603 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:06.039465 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:06.041057 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:06.041022 6275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5vjwt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:35:06.041184 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:06.041085 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" podUID="e7ec1529-3bf9-4ded-8a38-77e9b1b7964d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:35:06.087597 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:06.087485 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" podStartSLOduration=1.087471354 podStartE2EDuration="1.087471354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:06.086729985 +0000 UTC m=+334.478537612" watchObservedRunningTime="2026-10-30 01:35:06.087471354 +0000 UTC m=+334.479278987" Oct 30 01:35:07.043966 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.043933 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5vjwt" Oct 30 01:35:07.130909 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.130846 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp"] Oct 30 01:35:07.131205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.131086 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerName="discovery" containerID="cri-o://1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2" gracePeriod=30 Oct 30 01:35:07.374391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.374366 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:35:07.537996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.537950 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538044 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538073 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538102 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538131 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538170 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538208 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4\") pod \"d04b1405-2e5e-492c-8da4-94b0ae52267b\" (UID: \"d04b1405-2e5e-492c-8da4-94b0ae52267b\") " Oct 30 01:35:07.538618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.538587 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:07.540580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.540556 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "kube-api-access-xlgc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:07.540687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.540587 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:07.540800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.540783 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:07.540870 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.540799 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:07.540870 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.540779 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:07.540980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.540921 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert" pod "d04b1405-2e5e-492c-8da4-94b0ae52267b" (UID: "d04b1405-2e5e-492c-8da4-94b0ae52267b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639258 6275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-ca-configmap\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639290 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlgc4\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-kube-api-access-xlgc4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639300 6275 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-kubeconfig\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639310 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639319 6275 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-cacerts\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639326 6275 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04b1405-2e5e-492c-8da4-94b0ae52267b-local-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.639344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:07.639334 6275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d04b1405-2e5e-492c-8da4-94b0ae52267b-istio-csr-dns-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:35:08.048689 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.048653 6275 generic.go:356] "Generic (PLEG): container finished" podID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerID="1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2" exitCode=0 Oct 30 01:35:08.049103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.048704 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" Oct 30 01:35:08.049103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.048743 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" event={"ID":"d04b1405-2e5e-492c-8da4-94b0ae52267b","Type":"ContainerDied","Data":"1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2"} Oct 30 01:35:08.049103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.048797 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp" event={"ID":"d04b1405-2e5e-492c-8da4-94b0ae52267b","Type":"ContainerDied","Data":"0ac138b11c2524c6a8a1391c676cbee05db219b300027275b3b80dafc8e92ff7"} Oct 30 01:35:08.049103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.048820 6275 scope.go:122] "RemoveContainer" containerID="1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2" Oct 30 01:35:08.056580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.056567 6275 scope.go:122] "RemoveContainer" containerID="1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2" Oct 30 01:35:08.056829 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:35:08.056812 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2\": container with ID starting with 1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2 not found: ID does not exist" containerID="1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2" Oct 30 01:35:08.056867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.056837 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2"} err="failed to get container status \"1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2\": rpc error: code = NotFound desc = could not find container \"1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2\": container with ID starting with 1b14b134c197c8d04cb8e75d36eb3d168bf793c0ed59f367c6109d25f7e375c2 not found: ID does not exist" Oct 30 01:35:08.095575 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.095548 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp"] Oct 30 01:35:08.106169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:08.106145 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2gzsp"] Oct 30 01:35:09.736584 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:35:09.736544 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" path="/var/lib/kubelet/pods/d04b1405-2e5e-492c-8da4-94b0ae52267b/volumes" Oct 30 01:35:16.694896 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:35:16.694850 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:35:46.694313 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:35:46.694234 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:36:16.695371 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:36:16.695334 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:36:46.694476 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:36:46.694441 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:37:16.696030 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:37:16.695947 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:37:46.695224 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:37:46.695186 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:38:16.695428 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:38:16.695393 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:38:46.695376 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:38:46.695300 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:39:16.695189 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:39:16.695152 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:39:31.771185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:39:31.771158 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:39:31.771602 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:39:31.771365 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:39:46.697215 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:39:46.697176 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:40:16.695604 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:40:16.695527 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:40:46.694770 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:40:46.694734 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:41:16.696345 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:41:16.696299 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:41:46.695405 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:41:46.695329 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:42:16.695341 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:42:16.695299 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:42:46.694319 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:42:46.694280 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:43:16.695572 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:43:16.695491 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:43:46.694380 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:43:46.694340 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:44:16.694814 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:44:16.694781 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:44:31.795544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:31.795514 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:44:31.796439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:31.796413 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:44:37.957611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.957577 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:37.958039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.958019 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerName="discovery" Oct 30 01:44:37.958108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.958041 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerName="discovery" Oct 30 01:44:37.958199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.958142 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04b1405-2e5e-492c-8da4-94b0ae52267b" containerName="discovery" Oct 30 01:44:37.960999 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.960983 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:37.975861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.975837 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 01:44:37.975997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.975909 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 01:44:37.975997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.975990 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 01:44:37.986812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.986784 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w4cxk\"" Oct 30 01:44:37.987798 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.987774 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:37.997136 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.997098 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxv4w\" (UniqueName: \"kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w\") pod \"limitador-limitador-74dfcfb894-vz5pc\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:37.997263 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:37.997244 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file\") pod \"limitador-limitador-74dfcfb894-vz5pc\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:38.068168 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.068116 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:38.098332 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.098298 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxv4w\" (UniqueName: \"kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w\") pod \"limitador-limitador-74dfcfb894-vz5pc\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:38.098483 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.098353 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file\") pod \"limitador-limitador-74dfcfb894-vz5pc\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:38.098964 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.098945 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file\") pod \"limitador-limitador-74dfcfb894-vz5pc\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:38.120555 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.120525 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxv4w\" (UniqueName: \"kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w\") pod \"limitador-limitador-74dfcfb894-vz5pc\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:38.274640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.274611 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:38.571984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.571895 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:38.576841 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:44:38.576812 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd00a67_0d4d_49b6_8ed5_2c8115ebe48d.slice/crio-df15ad58b01b57fbfad992b50b6f1b18a090063adb6f62b5eaf051ad44a797fe WatchSource:0}: Error finding container df15ad58b01b57fbfad992b50b6f1b18a090063adb6f62b5eaf051ad44a797fe: Status 404 returned error can't find the container with id df15ad58b01b57fbfad992b50b6f1b18a090063adb6f62b5eaf051ad44a797fe Oct 30 01:44:38.578750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.578732 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:44:38.829185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.829093 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wtxbc"] Oct 30 01:44:38.833522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.833500 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:38.846965 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.846944 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k24md\"" Oct 30 01:44:38.851579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.851556 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wtxbc"] Oct 30 01:44:38.904425 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:38.904395 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsj8s\" (UniqueName: \"kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s\") pod \"authorino-57f7f8659f-wtxbc\" (UID: \"a5618bfb-cb21-4a68-a96e-486752cbc790\") " pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:39.005643 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.005609 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsj8s\" (UniqueName: \"kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s\") pod \"authorino-57f7f8659f-wtxbc\" (UID: \"a5618bfb-cb21-4a68-a96e-486752cbc790\") " pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:39.032217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.032184 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsj8s\" (UniqueName: \"kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s\") pod \"authorino-57f7f8659f-wtxbc\" (UID: \"a5618bfb-cb21-4a68-a96e-486752cbc790\") " pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:39.144917 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.144837 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:39.240535 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.240505 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" event={"ID":"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d","Type":"ContainerStarted","Data":"df15ad58b01b57fbfad992b50b6f1b18a090063adb6f62b5eaf051ad44a797fe"} Oct 30 01:44:39.325212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.325185 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7ctn"] Oct 30 01:44:39.328604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.328586 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:44:39.350717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.350688 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7ctn"] Oct 30 01:44:39.409444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.409370 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bg44\" (UniqueName: \"kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44\") pod \"authorino-56c6c9ddd5-z7ctn\" (UID: \"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0\") " pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:44:39.435453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.435421 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wtxbc"] Oct 30 01:44:39.440050 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:44:39.440020 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5618bfb_cb21_4a68_a96e_486752cbc790.slice/crio-601d9c7c5bf854ce9d5fd0e22c3c00bfcab278620436a736a86513495244fea3 WatchSource:0}: Error finding container 601d9c7c5bf854ce9d5fd0e22c3c00bfcab278620436a736a86513495244fea3: Status 404 returned error can't find the container with id 601d9c7c5bf854ce9d5fd0e22c3c00bfcab278620436a736a86513495244fea3 Oct 30 01:44:39.510809 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.510779 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bg44\" (UniqueName: \"kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44\") pod \"authorino-56c6c9ddd5-z7ctn\" (UID: \"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0\") " pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:44:39.548336 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.548307 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bg44\" (UniqueName: \"kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44\") pod \"authorino-56c6c9ddd5-z7ctn\" (UID: \"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0\") " pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:44:39.641944 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.641911 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:44:39.958939 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:39.958847 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7ctn"] Oct 30 01:44:39.963891 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:44:39.963852 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c02c219_0bc1_4c4f_b432_6b54bc92b2f0.slice/crio-3ac2081b293391ec7754f7d25cec0ddebdd51e7901f6c1b54507db593e46a325 WatchSource:0}: Error finding container 3ac2081b293391ec7754f7d25cec0ddebdd51e7901f6c1b54507db593e46a325: Status 404 returned error can't find the container with id 3ac2081b293391ec7754f7d25cec0ddebdd51e7901f6c1b54507db593e46a325 Oct 30 01:44:40.248780 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:40.248696 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" event={"ID":"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0","Type":"ContainerStarted","Data":"3ac2081b293391ec7754f7d25cec0ddebdd51e7901f6c1b54507db593e46a325"} Oct 30 01:44:40.250394 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:40.250357 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" event={"ID":"a5618bfb-cb21-4a68-a96e-486752cbc790","Type":"ContainerStarted","Data":"601d9c7c5bf854ce9d5fd0e22c3c00bfcab278620436a736a86513495244fea3"} Oct 30 01:44:42.291929 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.291888 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" event={"ID":"a5618bfb-cb21-4a68-a96e-486752cbc790","Type":"ContainerStarted","Data":"f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70"} Oct 30 01:44:42.293890 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.293819 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" event={"ID":"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d","Type":"ContainerStarted","Data":"33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d"} Oct 30 01:44:42.294143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.294089 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:42.296966 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.296945 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" event={"ID":"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0","Type":"ContainerStarted","Data":"350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b"} Oct 30 01:44:42.329148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.329051 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" podStartSLOduration=1.679730975 podStartE2EDuration="4.329039043s" totalImagesPullingTime="2.649308068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:42.328147193 +0000 UTC m=+910.719954817" watchObservedRunningTime="2026-10-30 01:44:42.329039043 +0000 UTC m=+910.720846693" Oct 30 01:44:42.370482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.370428 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" podStartSLOduration=2.211753323 podStartE2EDuration="5.370414576s" totalImagesPullingTime="3.158661253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:42.369799787 +0000 UTC m=+910.761607423" watchObservedRunningTime="2026-10-30 01:44:42.370414576 +0000 UTC m=+910.762222214" Oct 30 01:44:42.411824 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.411772 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" podStartSLOduration=1.296030474 podStartE2EDuration="3.411762292s" totalImagesPullingTime="2.115731818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:42.41069513 +0000 UTC m=+910.802502762" watchObservedRunningTime="2026-10-30 01:44:42.411762292 +0000 UTC m=+910.803569918" Oct 30 01:44:42.442073 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:42.442018 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wtxbc"] Oct 30 01:44:44.304985 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:44.304942 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" podUID="a5618bfb-cb21-4a68-a96e-486752cbc790" containerName="authorino" containerID="cri-o://f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70" gracePeriod=30 Oct 30 01:44:44.542383 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:44.542358 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:44.654353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:44.654260 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s\" (UniqueName: \"kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s\") pod \"a5618bfb-cb21-4a68-a96e-486752cbc790\" (UID: \"a5618bfb-cb21-4a68-a96e-486752cbc790\") " Oct 30 01:44:44.656466 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:44.656442 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s" pod "a5618bfb-cb21-4a68-a96e-486752cbc790" (UID: "a5618bfb-cb21-4a68-a96e-486752cbc790"). InnerVolumeSpecName "kube-api-access-dsj8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:44.755547 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:44.755509 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsj8s\" (UniqueName: \"kubernetes.io/projected/a5618bfb-cb21-4a68-a96e-486752cbc790-kube-api-access-dsj8s\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:44:45.310312 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.310279 6275 generic.go:356] "Generic (PLEG): container finished" podID="a5618bfb-cb21-4a68-a96e-486752cbc790" containerID="f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70" exitCode=0 Oct 30 01:44:45.310716 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.310321 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" Oct 30 01:44:45.310716 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.310337 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" event={"ID":"a5618bfb-cb21-4a68-a96e-486752cbc790","Type":"ContainerDied","Data":"f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70"} Oct 30 01:44:45.310716 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.310371 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wtxbc" event={"ID":"a5618bfb-cb21-4a68-a96e-486752cbc790","Type":"ContainerDied","Data":"601d9c7c5bf854ce9d5fd0e22c3c00bfcab278620436a736a86513495244fea3"} Oct 30 01:44:45.310716 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.310386 6275 scope.go:122] "RemoveContainer" containerID="f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70" Oct 30 01:44:45.319331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.319313 6275 scope.go:122] "RemoveContainer" containerID="f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70" Oct 30 01:44:45.319609 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:44:45.319588 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70\": container with ID starting with f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70 not found: ID does not exist" containerID="f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70" Oct 30 01:44:45.319671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.319619 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70"} err="failed to get container status \"f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70\": rpc error: code = NotFound desc = could not find container \"f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70\": container with ID starting with f10b6f1a2d9309a081c7f2fb37b1430236abd798728cbc66c2db4106ab689b70 not found: ID does not exist" Oct 30 01:44:45.370026 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.369997 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wtxbc"] Oct 30 01:44:45.374006 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.373981 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wtxbc"] Oct 30 01:44:45.736430 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:45.736347 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5618bfb-cb21-4a68-a96e-486752cbc790" path="/var/lib/kubelet/pods/a5618bfb-cb21-4a68-a96e-486752cbc790/volumes" Oct 30 01:44:46.694680 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:44:46.694631 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:44:52.090282 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.090246 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:52.090750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.090410 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" podUID="dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" containerName="limitador" containerID="cri-o://33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d" gracePeriod=30 Oct 30 01:44:52.091343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.091022 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:52.640664 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.640643 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:52.724431 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.724349 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file\" (UniqueName: \"kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file\") pod \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " Oct 30 01:44:52.724583 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.724486 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w\" (UniqueName: \"kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w\") pod \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\" (UID: \"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d\") " Oct 30 01:44:52.724775 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.724755 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file" pod "dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" (UID: "dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:52.726573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.726551 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w" pod "dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" (UID: "dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d"). InnerVolumeSpecName "kube-api-access-cxv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:52.825355 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.825320 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxv4w\" (UniqueName: \"kubernetes.io/projected/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-kube-api-access-cxv4w\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:44:52.825355 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:52.825348 6275 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d-config-file\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:44:53.347171 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.347107 6275 generic.go:356] "Generic (PLEG): container finished" podID="dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" containerID="33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d" exitCode=0 Oct 30 01:44:53.347748 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.347179 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" Oct 30 01:44:53.347748 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.347197 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" event={"ID":"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d","Type":"ContainerDied","Data":"33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d"} Oct 30 01:44:53.347748 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.347265 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vz5pc" event={"ID":"dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d","Type":"ContainerDied","Data":"df15ad58b01b57fbfad992b50b6f1b18a090063adb6f62b5eaf051ad44a797fe"} Oct 30 01:44:53.347748 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.347301 6275 scope.go:122] "RemoveContainer" containerID="33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d" Oct 30 01:44:53.355445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.355428 6275 scope.go:122] "RemoveContainer" containerID="33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d" Oct 30 01:44:53.355665 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:44:53.355647 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d\": container with ID starting with 33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d not found: ID does not exist" containerID="33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d" Oct 30 01:44:53.355734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.355678 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d"} err="failed to get container status \"33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d\": rpc error: code = NotFound desc = could not find container \"33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d\": container with ID starting with 33ae07131c6bcc42ca1c2ce4b6448a45cfee8e3bdd5882d15a1708cd0f15e85d not found: ID does not exist" Oct 30 01:44:53.394923 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.394886 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:53.404831 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.404803 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vz5pc"] Oct 30 01:44:53.736075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:53.736002 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" path="/var/lib/kubelet/pods/dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d/volumes" Oct 30 01:44:55.402410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402376 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x"] Oct 30 01:44:55.402799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402682 6275 cpu_manager.go:414] "containerMap: removing container" podUID="dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" containerName="limitador" Oct 30 01:44:55.402799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402693 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" containerName="limitador" Oct 30 01:44:55.402799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402740 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a5618bfb-cb21-4a68-a96e-486752cbc790" containerName="authorino" Oct 30 01:44:55.402799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402745 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5618bfb-cb21-4a68-a96e-486752cbc790" containerName="authorino" Oct 30 01:44:55.402799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402799 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5618bfb-cb21-4a68-a96e-486752cbc790" containerName="authorino" Oct 30 01:44:55.403105 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.402813 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd00a67-0d4d-49b6-8ed5-2c8115ebe48d" containerName="limitador" Oct 30 01:44:55.407195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.407173 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.431069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.431039 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:55.436608 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.436582 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:55.436608 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.436601 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 01:44:55.436786 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.436665 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-r2cbw\"" Oct 30 01:44:55.441790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.441765 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x"] Oct 30 01:44:55.547681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.547646 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf-data\") pod \"postgres-7cdbf6dc44-b7p6x\" (UID: \"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.547681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.547683 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwt4f\" (UniqueName: \"kubernetes.io/projected/7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf-kube-api-access-vwt4f\") pod \"postgres-7cdbf6dc44-b7p6x\" (UID: \"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.648542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.648505 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf-data\") pod \"postgres-7cdbf6dc44-b7p6x\" (UID: \"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.648542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.648548 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwt4f\" (UniqueName: \"kubernetes.io/projected/7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf-kube-api-access-vwt4f\") pod \"postgres-7cdbf6dc44-b7p6x\" (UID: \"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.648904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.648880 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf-data\") pod \"postgres-7cdbf6dc44-b7p6x\" (UID: \"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.673135 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.673071 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwt4f\" (UniqueName: \"kubernetes.io/projected/7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf-kube-api-access-vwt4f\") pod \"postgres-7cdbf6dc44-b7p6x\" (UID: \"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:55.719195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:55.719173 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:44:56.019041 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:56.019009 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x"] Oct 30 01:44:56.023183 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:44:56.023156 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbd9e1d_6767_4ca5_89d0_aa99a2c89ebf.slice/crio-32591185c143001f7192a0b5537876e48b30f875b898ddb52375fb62047448f0 WatchSource:0}: Error finding container 32591185c143001f7192a0b5537876e48b30f875b898ddb52375fb62047448f0: Status 404 returned error can't find the container with id 32591185c143001f7192a0b5537876e48b30f875b898ddb52375fb62047448f0 Oct 30 01:44:56.360839 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:44:56.360755 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" event={"ID":"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf","Type":"ContainerStarted","Data":"32591185c143001f7192a0b5537876e48b30f875b898ddb52375fb62047448f0"} Oct 30 01:45:01.384474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:01.384430 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" event={"ID":"7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf","Type":"ContainerStarted","Data":"b4520fa5e5087209ddf67a0f1c8a8e5dc41eb65dc652f9de82179744e79cfd2e"} Oct 30 01:45:01.384886 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:01.384519 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:45:07.417541 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:07.417515 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" Oct 30 01:45:07.455656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:07.455611 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7p6x" podStartSLOduration=7.59528336 podStartE2EDuration="12.455599403s" totalImagesPullingTime="4.860316043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:01.425971937 +0000 UTC m=+929.817779571" watchObservedRunningTime="2026-10-30 01:45:07.455599403 +0000 UTC m=+935.847407072" Oct 30 01:45:14.289726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.289695 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6579864b68-5hcbw"] Oct 30 01:45:14.311216 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.311189 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.317729 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.317694 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6579864b68-5hcbw"] Oct 30 01:45:14.327942 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.327918 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 01:45:14.327942 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.327918 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 01:45:14.333346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.333323 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 01:45:14.338205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.338189 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pm2l8\"" Oct 30 01:45:14.338416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.338401 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 01:45:14.338473 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.338423 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 01:45:14.403481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.403444 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zjv\" (UniqueName: \"kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.403660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.403497 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.403660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.403616 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.504760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.504719 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.504936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.504775 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.504936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.504823 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zjv\" (UniqueName: \"kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.507221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.507190 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.507338 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.507233 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.526356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.526329 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zjv\" (UniqueName: \"kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv\") pod \"maas-controller-6579864b68-5hcbw\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.624365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.624273 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:14.801313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.801279 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75b4575b56-7wgng"] Oct 30 01:45:14.804592 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.804567 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:14.826702 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.826677 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75b4575b56-7wgng"] Oct 30 01:45:14.908893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.908807 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6n5\" (UniqueName: \"kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:14.908893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.908860 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:14.908893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.908883 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:14.919674 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:14.919645 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6579864b68-5hcbw"] Oct 30 01:45:14.924320 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:45:14.924290 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b48e1a_aaef_49d5_a6ed_d5998fb96729.slice/crio-058df9ee796d2bb657261aa8fd315be7ad2ae6ec912e803a93f21f00c1472aa8 WatchSource:0}: Error finding container 058df9ee796d2bb657261aa8fd315be7ad2ae6ec912e803a93f21f00c1472aa8: Status 404 returned error can't find the container with id 058df9ee796d2bb657261aa8fd315be7ad2ae6ec912e803a93f21f00c1472aa8 Oct 30 01:45:15.009917 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.009877 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6n5\" (UniqueName: \"kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.010090 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.009939 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.010090 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.009965 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.012477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.012453 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.012594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.012495 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.031670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.031643 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6n5\" (UniqueName: \"kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5\") pod \"maas-controller-75b4575b56-7wgng\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.115614 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.115579 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:15.414362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.414259 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75b4575b56-7wgng"] Oct 30 01:45:15.419089 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:45:15.419058 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400efcf6_470e_4589_9ad0_fb0c4458116b.slice/crio-a3aef0c1ba02c4a74bd62d0b8ddc90374b87ebc98d79937f4c203d5ad6143e08 WatchSource:0}: Error finding container a3aef0c1ba02c4a74bd62d0b8ddc90374b87ebc98d79937f4c203d5ad6143e08: Status 404 returned error can't find the container with id a3aef0c1ba02c4a74bd62d0b8ddc90374b87ebc98d79937f4c203d5ad6143e08 Oct 30 01:45:15.444193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.444153 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6579864b68-5hcbw" event={"ID":"f3b48e1a-aaef-49d5-a6ed-d5998fb96729","Type":"ContainerStarted","Data":"058df9ee796d2bb657261aa8fd315be7ad2ae6ec912e803a93f21f00c1472aa8"} Oct 30 01:45:15.445526 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:15.445495 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-7wgng" event={"ID":"400efcf6-470e-4589-9ad0-fb0c4458116b","Type":"ContainerStarted","Data":"a3aef0c1ba02c4a74bd62d0b8ddc90374b87ebc98d79937f4c203d5ad6143e08"} Oct 30 01:45:16.702083 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:45:16.702047 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:45:18.461551 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:18.461511 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-7wgng" event={"ID":"400efcf6-470e-4589-9ad0-fb0c4458116b","Type":"ContainerStarted","Data":"a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9"} Oct 30 01:45:18.462009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:18.461610 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:18.463004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:18.462978 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6579864b68-5hcbw" event={"ID":"f3b48e1a-aaef-49d5-a6ed-d5998fb96729","Type":"ContainerStarted","Data":"aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0"} Oct 30 01:45:18.463106 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:18.463036 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:18.510474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:18.510415 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75b4575b56-7wgng" podStartSLOduration=1.9990467440000002 podStartE2EDuration="4.510399976s" totalImagesPullingTime="2.511353232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:18.509493751 +0000 UTC m=+946.901301385" watchObservedRunningTime="2026-10-30 01:45:18.510399976 +0000 UTC m=+946.902207660" Oct 30 01:45:18.552155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:18.552089 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6579864b68-5hcbw" podStartSLOduration=1.854821516 podStartE2EDuration="4.55207521s" totalImagesPullingTime="2.697253694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:18.55149638 +0000 UTC m=+946.943304014" watchObservedRunningTime="2026-10-30 01:45:18.55207521 +0000 UTC m=+946.943882834" Oct 30 01:45:19.772013 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.771980 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d"] Oct 30 01:45:19.774657 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.774634 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.787823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.787797 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 01:45:19.798335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.798307 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 01:45:19.798485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.798340 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 01:45:19.798485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.798384 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-wtk2v\"" Oct 30 01:45:19.800210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.800183 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d"] Oct 30 01:45:19.852285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.852252 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ac253b0d-c298-4218-93c7-8a8787bfe66c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.852285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.852285 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac253b0d-c298-4218-93c7-8a8787bfe66c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.852495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.852309 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt676\" (UniqueName: \"kubernetes.io/projected/ac253b0d-c298-4218-93c7-8a8787bfe66c-kube-api-access-xt676\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.852495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.852359 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac253b0d-c298-4218-93c7-8a8787bfe66c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.953410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.953376 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac253b0d-c298-4218-93c7-8a8787bfe66c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.953600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.953456 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ac253b0d-c298-4218-93c7-8a8787bfe66c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.953600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.953474 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac253b0d-c298-4218-93c7-8a8787bfe66c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.953600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.953494 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt676\" (UniqueName: \"kubernetes.io/projected/ac253b0d-c298-4218-93c7-8a8787bfe66c-kube-api-access-xt676\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.954191 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.954173 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac253b0d-c298-4218-93c7-8a8787bfe66c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.954407 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.954387 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ac253b0d-c298-4218-93c7-8a8787bfe66c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.955861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.955838 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac253b0d-c298-4218-93c7-8a8787bfe66c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:19.975163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:19.975140 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt676\" (UniqueName: \"kubernetes.io/projected/ac253b0d-c298-4218-93c7-8a8787bfe66c-kube-api-access-xt676\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6sv2d\" (UID: \"ac253b0d-c298-4218-93c7-8a8787bfe66c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:20.085781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:20.085679 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:20.387290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:20.387195 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d"] Oct 30 01:45:20.392540 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:45:20.392499 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac253b0d_c298_4218_93c7_8a8787bfe66c.slice/crio-fec785b61bb64158dcd83c550e87f752c80cb91def01b35578a1313a3f3f8c5f WatchSource:0}: Error finding container fec785b61bb64158dcd83c550e87f752c80cb91def01b35578a1313a3f3f8c5f: Status 404 returned error can't find the container with id fec785b61bb64158dcd83c550e87f752c80cb91def01b35578a1313a3f3f8c5f Oct 30 01:45:20.472843 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:20.472809 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" event={"ID":"ac253b0d-c298-4218-93c7-8a8787bfe66c","Type":"ContainerStarted","Data":"fec785b61bb64158dcd83c550e87f752c80cb91def01b35578a1313a3f3f8c5f"} Oct 30 01:45:29.469757 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.469722 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 01:45:29.470232 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.469802 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:29.542039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.541243 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6579864b68-5hcbw"] Oct 30 01:45:29.542039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.541429 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6579864b68-5hcbw" podUID="f3b48e1a-aaef-49d5-a6ed-d5998fb96729" containerName="manager" containerID="cri-o://aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0" gracePeriod=30 Oct 30 01:45:29.796411 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.796381 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:29.839201 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.839165 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs\") pod \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " Oct 30 01:45:29.839392 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.839282 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert\") pod \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " Oct 30 01:45:29.839392 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.839332 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv\" (UniqueName: \"kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv\") pod \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\" (UID: \"f3b48e1a-aaef-49d5-a6ed-d5998fb96729\") " Oct 30 01:45:29.842455 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.842356 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv" pod "f3b48e1a-aaef-49d5-a6ed-d5998fb96729" (UID: "f3b48e1a-aaef-49d5-a6ed-d5998fb96729"). InnerVolumeSpecName "kube-api-access-c2zjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:29.844224 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.844197 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs" pod "f3b48e1a-aaef-49d5-a6ed-d5998fb96729" (UID: "f3b48e1a-aaef-49d5-a6ed-d5998fb96729"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:29.844335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.844290 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert" pod "f3b48e1a-aaef-49d5-a6ed-d5998fb96729" (UID: "f3b48e1a-aaef-49d5-a6ed-d5998fb96729"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:29.940521 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.940485 6275 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-webhook-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:45:29.940521 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.940522 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zjv\" (UniqueName: \"kubernetes.io/projected/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-kube-api-access-c2zjv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:45:29.940732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:29.940535 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b48e1a-aaef-49d5-a6ed-d5998fb96729-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:45:30.521809 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.521772 6275 generic.go:356] "Generic (PLEG): container finished" podID="f3b48e1a-aaef-49d5-a6ed-d5998fb96729" containerID="aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0" exitCode=0 Oct 30 01:45:30.522290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.521826 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6579864b68-5hcbw" Oct 30 01:45:30.522290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.521857 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6579864b68-5hcbw" event={"ID":"f3b48e1a-aaef-49d5-a6ed-d5998fb96729","Type":"ContainerDied","Data":"aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0"} Oct 30 01:45:30.522290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.521911 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6579864b68-5hcbw" event={"ID":"f3b48e1a-aaef-49d5-a6ed-d5998fb96729","Type":"ContainerDied","Data":"058df9ee796d2bb657261aa8fd315be7ad2ae6ec912e803a93f21f00c1472aa8"} Oct 30 01:45:30.522290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.521932 6275 scope.go:122] "RemoveContainer" containerID="aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0" Oct 30 01:45:30.531708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.531691 6275 scope.go:122] "RemoveContainer" containerID="aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0" Oct 30 01:45:30.531969 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:45:30.531950 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0\": container with ID starting with aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0 not found: ID does not exist" containerID="aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0" Oct 30 01:45:30.532032 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.531983 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0"} err="failed to get container status \"aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0\": rpc error: code = NotFound desc = could not find container \"aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0\": container with ID starting with aa286fa1391b40aa4d99981688e4ce8ee5d546a242f7fc2c843b2c365e450ec0 not found: ID does not exist" Oct 30 01:45:30.567311 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.567283 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6579864b68-5hcbw"] Oct 30 01:45:30.578007 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:30.577983 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6579864b68-5hcbw"] Oct 30 01:45:31.527862 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:31.527827 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" event={"ID":"ac253b0d-c298-4218-93c7-8a8787bfe66c","Type":"ContainerStarted","Data":"1d2b578101c3441f9ccdab365582a5dab124c50c8a8108e9367136c95f380b90"} Oct 30 01:45:31.528368 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:31.527969 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:31.566561 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:31.566516 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" podStartSLOduration=2.331542542 podStartE2EDuration="12.566502617s" totalImagesPullingTime="10.234960075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:31.566345981 +0000 UTC m=+959.958153615" watchObservedRunningTime="2026-10-30 01:45:31.566502617 +0000 UTC m=+959.958310249" Oct 30 01:45:31.737254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:31.737225 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b48e1a-aaef-49d5-a6ed-d5998fb96729" path="/var/lib/kubelet/pods/f3b48e1a-aaef-49d5-a6ed-d5998fb96729/volumes" Oct 30 01:45:42.538046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:42.538014 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6sv2d" Oct 30 01:45:46.695382 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:45:46.695345 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:45:50.804225 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.804192 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4"] Oct 30 01:45:50.804652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.804619 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f3b48e1a-aaef-49d5-a6ed-d5998fb96729" containerName="manager" Oct 30 01:45:50.804652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.804636 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b48e1a-aaef-49d5-a6ed-d5998fb96729" containerName="manager" Oct 30 01:45:50.804750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.804721 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b48e1a-aaef-49d5-a6ed-d5998fb96729" containerName="manager" Oct 30 01:45:50.811798 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.811774 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:50.835378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.835322 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xzt6k\"" Oct 30 01:45:50.835378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.835355 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 01:45:50.835558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.835439 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:45:50.839550 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.839526 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4"] Oct 30 01:45:50.922380 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.922348 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:50.922548 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.922395 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:50.922548 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.922423 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mt4\" (UniqueName: \"kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:50.922620 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:50.922558 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.023373 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.023339 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.023575 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.023381 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.023575 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.023406 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mt4\" (UniqueName: \"kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.023575 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.023429 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.024099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.024073 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.025878 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.025852 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.026059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.026033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.044799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.044778 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mt4\" (UniqueName: \"kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4\") pod \"maas-api-7d7b7b8cdf-x7kq4\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.126660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.126584 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:51.473051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.472916 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4"] Oct 30 01:45:51.477229 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:45:51.477198 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2568d4_394c_4bbf_8d34_b700bbfe514f.slice/crio-354e5a0ed774fe5189b16a7a94c3a37805fd0a135f0bfbcff8e3c6ca747c8f59 WatchSource:0}: Error finding container 354e5a0ed774fe5189b16a7a94c3a37805fd0a135f0bfbcff8e3c6ca747c8f59: Status 404 returned error can't find the container with id 354e5a0ed774fe5189b16a7a94c3a37805fd0a135f0bfbcff8e3c6ca747c8f59 Oct 30 01:45:51.607583 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:51.607545 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" event={"ID":"cb2568d4-394c-4bbf-8d34-b700bbfe514f","Type":"ContainerStarted","Data":"354e5a0ed774fe5189b16a7a94c3a37805fd0a135f0bfbcff8e3c6ca747c8f59"} Oct 30 01:45:54.627894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:54.627849 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" event={"ID":"cb2568d4-394c-4bbf-8d34-b700bbfe514f","Type":"ContainerStarted","Data":"9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1"} Oct 30 01:45:54.628346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:54.627975 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:45:54.670907 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:45:54.670852 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" podStartSLOduration=2.244515494 podStartE2EDuration="4.670838377s" totalImagesPullingTime="2.426322883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:54.670082613 +0000 UTC m=+983.061890245" watchObservedRunningTime="2026-10-30 01:45:54.670838377 +0000 UTC m=+983.062646010" Oct 30 01:46:00.637825 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:00.637794 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:46:01.396415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.396383 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bzgvb"] Oct 30 01:46:01.409834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.409809 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:01.419372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.419348 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bzgvb"] Oct 30 01:46:01.516270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.516233 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86r77\" (UniqueName: \"kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77\") pod \"authorino-6f6944bcdd-bzgvb\" (UID: \"540c9d60-38b6-420b-aa85-402130d10f9c\") " pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:01.579609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.579556 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2"] Oct 30 01:46:01.582048 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.582025 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.607493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.607460 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2"] Oct 30 01:46:01.617666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.617632 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86r77\" (UniqueName: \"kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77\") pod \"authorino-6f6944bcdd-bzgvb\" (UID: \"540c9d60-38b6-420b-aa85-402130d10f9c\") " pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:01.642394 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.642364 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86r77\" (UniqueName: \"kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77\") pod \"authorino-6f6944bcdd-bzgvb\" (UID: \"540c9d60-38b6-420b-aa85-402130d10f9c\") " pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:01.671784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.671694 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bzgvb"] Oct 30 01:46:01.671952 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.671934 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:01.698151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.698110 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w8xm8"] Oct 30 01:46:01.701286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.701266 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.715972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.715944 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 01:46:01.718097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.718072 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.718211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.718148 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55925\" (UniqueName: \"kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.718293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.718271 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.718349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.718331 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.723730 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.723700 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w8xm8"] Oct 30 01:46:01.753589 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.753557 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w8xm8"] Oct 30 01:46:01.753899 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:46:01.753873 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fn4k4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" podUID="51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7" Oct 30 01:46:01.780049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.780010 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59cdc7947-244dk"] Oct 30 01:46:01.782526 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.782498 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:01.803699 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.803676 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59cdc7947-244dk"] Oct 30 01:46:01.819366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.819194 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55925\" (UniqueName: \"kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.819366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.819253 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.819366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.819282 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4k4\" (UniqueName: \"kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4\") pod \"authorino-6889cbd8c4-w8xm8\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.819585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.819481 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.819585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.819562 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.819692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.819590 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert\") pod \"authorino-6889cbd8c4-w8xm8\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.820225 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.820202 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.821894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.821868 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.822081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.822064 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.843060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.843034 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55925\" (UniqueName: \"kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925\") pod \"maas-api-647d7b7446-s6tp2\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.893360 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.893330 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:01.920492 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.920462 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert\") pod \"authorino-6889cbd8c4-w8xm8\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.920648 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.920520 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdbs\" (UniqueName: \"kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs\") pod \"authorino-59cdc7947-244dk\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:01.920648 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.920564 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4k4\" (UniqueName: \"kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4\") pod \"authorino-6889cbd8c4-w8xm8\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.920648 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.920615 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert\") pod \"authorino-59cdc7947-244dk\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:01.922894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.922845 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert\") pod \"authorino-6889cbd8c4-w8xm8\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.943322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.943295 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4k4\" (UniqueName: \"kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4\") pod \"authorino-6889cbd8c4-w8xm8\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:01.969033 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:01.968999 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bzgvb"] Oct 30 01:46:01.974357 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:01.974326 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod540c9d60_38b6_420b_aa85_402130d10f9c.slice/crio-707ceb3b6d107c5aae7733b5655e848dade7f671cd412eb10e53713bf3bc5438 WatchSource:0}: Error finding container 707ceb3b6d107c5aae7733b5655e848dade7f671cd412eb10e53713bf3bc5438: Status 404 returned error can't find the container with id 707ceb3b6d107c5aae7733b5655e848dade7f671cd412eb10e53713bf3bc5438 Oct 30 01:46:02.021683 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.021649 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdbs\" (UniqueName: \"kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs\") pod \"authorino-59cdc7947-244dk\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:02.021851 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.021710 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert\") pod \"authorino-59cdc7947-244dk\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:02.024160 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.024116 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert\") pod \"authorino-59cdc7947-244dk\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:02.044356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.044330 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdbs\" (UniqueName: \"kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs\") pod \"authorino-59cdc7947-244dk\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:02.093345 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.093310 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:46:02.192871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.192788 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2"] Oct 30 01:46:02.196707 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:02.196685 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96f255f_31d7_4bc1_b476_b4e01f0018ca.slice/crio-97bc18e7fc8870304b9cdf055db6c261772335348041c17e99994420f322bc02 WatchSource:0}: Error finding container 97bc18e7fc8870304b9cdf055db6c261772335348041c17e99994420f322bc02: Status 404 returned error can't find the container with id 97bc18e7fc8870304b9cdf055db6c261772335348041c17e99994420f322bc02 Oct 30 01:46:02.388157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.388104 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59cdc7947-244dk"] Oct 30 01:46:02.404062 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:02.404032 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ef337d6_2b13_4ff8_8a55_333a6cdae799.slice/crio-7245be81dd5dfbc8d37ff0eddcd4b863af09a18971621a10eef0e08be73b9aec WatchSource:0}: Error finding container 7245be81dd5dfbc8d37ff0eddcd4b863af09a18971621a10eef0e08be73b9aec: Status 404 returned error can't find the container with id 7245be81dd5dfbc8d37ff0eddcd4b863af09a18971621a10eef0e08be73b9aec Oct 30 01:46:02.664402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.664357 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cdc7947-244dk" event={"ID":"6ef337d6-2b13-4ff8-8a55-333a6cdae799","Type":"ContainerStarted","Data":"7245be81dd5dfbc8d37ff0eddcd4b863af09a18971621a10eef0e08be73b9aec"} Oct 30 01:46:02.666039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.666004 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" event={"ID":"f96f255f-31d7-4bc1-b476-b4e01f0018ca","Type":"ContainerStarted","Data":"966ef45a3a6edcee192b6ec57bb8b6c894b0cd67822612e27b7ec8429ba80bfd"} Oct 30 01:46:02.666197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.666064 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" event={"ID":"f96f255f-31d7-4bc1-b476-b4e01f0018ca","Type":"ContainerStarted","Data":"97bc18e7fc8870304b9cdf055db6c261772335348041c17e99994420f322bc02"} Oct 30 01:46:02.666197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.666115 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:02.667277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.667257 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" event={"ID":"540c9d60-38b6-420b-aa85-402130d10f9c","Type":"ContainerStarted","Data":"707ceb3b6d107c5aae7733b5655e848dade7f671cd412eb10e53713bf3bc5438"} Oct 30 01:46:02.667277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.667274 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:02.674296 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.674276 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:02.705864 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.705819 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" podStartSLOduration=1.43574377 podStartE2EDuration="1.705805431s" totalImagesPullingTime="270.061661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:02.704853736 +0000 UTC m=+991.096661369" watchObservedRunningTime="2026-10-30 01:46:02.705805431 +0000 UTC m=+991.097613101" Oct 30 01:46:02.830072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.830037 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4\" (UniqueName: \"kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4\") pod \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " Oct 30 01:46:02.830326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.830292 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert\" (UniqueName: \"kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert\") pod \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\" (UID: \"51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7\") " Oct 30 01:46:02.832227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.832193 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4" pod "51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7" (UID: "51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7"). InnerVolumeSpecName "kube-api-access-fn4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:02.832347 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.832249 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert" pod "51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7" (UID: "51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:02.932022 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.931993 6275 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-tls-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:02.932022 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:02.932024 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4k4\" (UniqueName: \"kubernetes.io/projected/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7-kube-api-access-fn4k4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:03.672432 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.672394 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cdc7947-244dk" event={"ID":"6ef337d6-2b13-4ff8-8a55-333a6cdae799","Type":"ContainerStarted","Data":"b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5"} Oct 30 01:46:03.673917 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.673886 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" event={"ID":"540c9d60-38b6-420b-aa85-402130d10f9c","Type":"ContainerStarted","Data":"510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11"} Oct 30 01:46:03.674009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.673911 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" podUID="540c9d60-38b6-420b-aa85-402130d10f9c" containerName="authorino" containerID="cri-o://510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11" gracePeriod=30 Oct 30 01:46:03.674083 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.674013 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w8xm8" Oct 30 01:46:03.708990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.708941 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59cdc7947-244dk" podStartSLOduration=2.1998507099999998 podStartE2EDuration="2.708927157s" totalImagesPullingTime="509.076447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:03.708023057 +0000 UTC m=+992.099830688" watchObservedRunningTime="2026-10-30 01:46:03.708927157 +0000 UTC m=+992.100734800" Oct 30 01:46:03.739187 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.739159 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7ctn"] Oct 30 01:46:03.739384 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.739363 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" podUID="6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" containerName="authorino" containerID="cri-o://350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b" gracePeriod=30 Oct 30 01:46:03.745696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.745561 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" podStartSLOduration=1.878896669 podStartE2EDuration="2.745546617s" totalImagesPullingTime="866.649948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:03.744024478 +0000 UTC m=+992.135832112" watchObservedRunningTime="2026-10-30 01:46:03.745546617 +0000 UTC m=+992.137354251" Oct 30 01:46:03.799542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.799509 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w8xm8"] Oct 30 01:46:03.804658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.804628 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w8xm8"] Oct 30 01:46:03.992173 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.992151 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:03.994931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:03.994914 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:46:04.143534 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.143500 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44\" (UniqueName: \"kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44\") pod \"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0\" (UID: \"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0\") " Oct 30 01:46:04.143684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.143602 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77\" (UniqueName: \"kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77\") pod \"540c9d60-38b6-420b-aa85-402130d10f9c\" (UID: \"540c9d60-38b6-420b-aa85-402130d10f9c\") " Oct 30 01:46:04.145571 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.145535 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44" pod "6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" (UID: "6c02c219-0bc1-4c4f-b432-6b54bc92b2f0"). InnerVolumeSpecName "kube-api-access-6bg44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:04.145701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.145592 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77" pod "540c9d60-38b6-420b-aa85-402130d10f9c" (UID: "540c9d60-38b6-420b-aa85-402130d10f9c"). InnerVolumeSpecName "kube-api-access-86r77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:04.245267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.245186 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bg44\" (UniqueName: \"kubernetes.io/projected/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0-kube-api-access-6bg44\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:04.245267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.245215 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86r77\" (UniqueName: \"kubernetes.io/projected/540c9d60-38b6-420b-aa85-402130d10f9c-kube-api-access-86r77\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:04.678848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.678810 6275 generic.go:356] "Generic (PLEG): container finished" podID="540c9d60-38b6-420b-aa85-402130d10f9c" containerID="510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11" exitCode=0 Oct 30 01:46:04.679290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.678857 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" Oct 30 01:46:04.679290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.678900 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" event={"ID":"540c9d60-38b6-420b-aa85-402130d10f9c","Type":"ContainerDied","Data":"510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11"} Oct 30 01:46:04.679290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.678944 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-bzgvb" event={"ID":"540c9d60-38b6-420b-aa85-402130d10f9c","Type":"ContainerDied","Data":"707ceb3b6d107c5aae7733b5655e848dade7f671cd412eb10e53713bf3bc5438"} Oct 30 01:46:04.679290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.678965 6275 scope.go:122] "RemoveContainer" containerID="510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11" Oct 30 01:46:04.680195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.680171 6275 generic.go:356] "Generic (PLEG): container finished" podID="6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" containerID="350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b" exitCode=0 Oct 30 01:46:04.680298 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.680227 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" Oct 30 01:46:04.680298 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.680255 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" event={"ID":"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0","Type":"ContainerDied","Data":"350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b"} Oct 30 01:46:04.680298 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.680295 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7ctn" event={"ID":"6c02c219-0bc1-4c4f-b432-6b54bc92b2f0","Type":"ContainerDied","Data":"3ac2081b293391ec7754f7d25cec0ddebdd51e7901f6c1b54507db593e46a325"} Oct 30 01:46:04.686892 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.686864 6275 scope.go:122] "RemoveContainer" containerID="510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11" Oct 30 01:46:04.687239 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:46:04.687215 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11\": container with ID starting with 510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11 not found: ID does not exist" containerID="510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11" Oct 30 01:46:04.687342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.687247 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11"} err="failed to get container status \"510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11\": rpc error: code = NotFound desc = could not find container \"510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11\": container with ID starting with 510b5267c2b603a39d4a6205e2a8e61e62563bcad42307360daf9c2cafd76b11 not found: ID does not exist" Oct 30 01:46:04.687342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.687268 6275 scope.go:122] "RemoveContainer" containerID="350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b" Oct 30 01:46:04.694897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.694880 6275 scope.go:122] "RemoveContainer" containerID="350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b" Oct 30 01:46:04.695170 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:46:04.695150 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b\": container with ID starting with 350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b not found: ID does not exist" containerID="350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b" Oct 30 01:46:04.695228 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.695177 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b"} err="failed to get container status \"350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b\": rpc error: code = NotFound desc = could not find container \"350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b\": container with ID starting with 350d13e2eb3e1bc338c4c98dbcbbb76ac23bcc25e36316a8f50575e99a095e5b not found: ID does not exist" Oct 30 01:46:04.723111 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.723091 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7ctn"] Oct 30 01:46:04.726630 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.726610 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7ctn"] Oct 30 01:46:04.755802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.755780 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bzgvb"] Oct 30 01:46:04.758089 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:04.758068 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bzgvb"] Oct 30 01:46:05.736105 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:05.736067 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7" path="/var/lib/kubelet/pods/51b4ba0a-9c3c-45ff-80e1-8b64dabae2a7/volumes" Oct 30 01:46:05.736634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:05.736385 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540c9d60-38b6-420b-aa85-402130d10f9c" path="/var/lib/kubelet/pods/540c9d60-38b6-420b-aa85-402130d10f9c/volumes" Oct 30 01:46:05.736799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:05.736780 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" path="/var/lib/kubelet/pods/6c02c219-0bc1-4c4f-b432-6b54bc92b2f0/volumes" Oct 30 01:46:08.679499 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:08.679472 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 01:46:08.752384 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:08.752106 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4"] Oct 30 01:46:08.752666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:08.752636 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" podUID="cb2568d4-394c-4bbf-8d34-b700bbfe514f" containerName="maas-api" containerID="cri-o://9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1" gracePeriod=30 Oct 30 01:46:08.995634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:08.995605 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:46:09.183858 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.183819 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs\") pod \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " Oct 30 01:46:09.184035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.183930 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4\" (UniqueName: \"kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4\") pod \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " Oct 30 01:46:09.184035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.183967 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca\") pod \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " Oct 30 01:46:09.184035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.184006 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls\") pod \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\" (UID: \"cb2568d4-394c-4bbf-8d34-b700bbfe514f\") " Oct 30 01:46:09.184350 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.184315 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca" pod "cb2568d4-394c-4bbf-8d34-b700bbfe514f" (UID: "cb2568d4-394c-4bbf-8d34-b700bbfe514f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:09.186295 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.186260 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls" pod "cb2568d4-394c-4bbf-8d34-b700bbfe514f" (UID: "cb2568d4-394c-4bbf-8d34-b700bbfe514f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:09.186403 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.186305 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs" pod "cb2568d4-394c-4bbf-8d34-b700bbfe514f" (UID: "cb2568d4-394c-4bbf-8d34-b700bbfe514f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:09.186403 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.186360 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4" pod "cb2568d4-394c-4bbf-8d34-b700bbfe514f" (UID: "cb2568d4-394c-4bbf-8d34-b700bbfe514f"). InnerVolumeSpecName "kube-api-access-s6mt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:09.284656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.284629 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:09.284656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.284656 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mt4\" (UniqueName: \"kubernetes.io/projected/cb2568d4-394c-4bbf-8d34-b700bbfe514f-kube-api-access-s6mt4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:09.284842 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.284665 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2568d4-394c-4bbf-8d34-b700bbfe514f-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:09.284842 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.284676 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2568d4-394c-4bbf-8d34-b700bbfe514f-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:46:09.709378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.709279 6275 generic.go:356] "Generic (PLEG): container finished" podID="cb2568d4-394c-4bbf-8d34-b700bbfe514f" containerID="9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1" exitCode=0 Oct 30 01:46:09.709378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.709359 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" Oct 30 01:46:09.709378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.709363 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" event={"ID":"cb2568d4-394c-4bbf-8d34-b700bbfe514f","Type":"ContainerDied","Data":"9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1"} Oct 30 01:46:09.709980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.709449 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4" event={"ID":"cb2568d4-394c-4bbf-8d34-b700bbfe514f","Type":"ContainerDied","Data":"354e5a0ed774fe5189b16a7a94c3a37805fd0a135f0bfbcff8e3c6ca747c8f59"} Oct 30 01:46:09.709980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.709474 6275 scope.go:122] "RemoveContainer" containerID="9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1" Oct 30 01:46:09.718019 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.718003 6275 scope.go:122] "RemoveContainer" containerID="9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1" Oct 30 01:46:09.718299 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:46:09.718281 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1\": container with ID starting with 9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1 not found: ID does not exist" containerID="9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1" Oct 30 01:46:09.718346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.718310 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1"} err="failed to get container status \"9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1\": rpc error: code = NotFound desc = could not find container \"9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1\": container with ID starting with 9baae6505d1fb91c4d4d8934d587fac0f6413a144b9a0b39032e629f882a62f1 not found: ID does not exist" Oct 30 01:46:09.756459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.756426 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4"] Oct 30 01:46:09.767055 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:09.767018 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d7b7b8cdf-x7kq4"] Oct 30 01:46:11.735977 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:11.735939 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2568d4-394c-4bbf-8d34-b700bbfe514f" path="/var/lib/kubelet/pods/cb2568d4-394c-4bbf-8d34-b700bbfe514f/volumes" Oct 30 01:46:16.696064 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:46:16.696023 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:46:38.944244 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944211 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pd2tp"] Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944552 6275 cpu_manager.go:414] "containerMap: removing container" podUID="cb2568d4-394c-4bbf-8d34-b700bbfe514f" containerName="maas-api" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944564 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2568d4-394c-4bbf-8d34-b700bbfe514f" containerName="maas-api" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944579 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" containerName="authorino" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944587 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" containerName="authorino" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944608 6275 cpu_manager.go:414] "containerMap: removing container" podUID="540c9d60-38b6-420b-aa85-402130d10f9c" containerName="authorino" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944614 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540c9d60-38b6-420b-aa85-402130d10f9c" containerName="authorino" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944690 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2568d4-394c-4bbf-8d34-b700bbfe514f" containerName="maas-api" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944701 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c02c219-0bc1-4c4f-b432-6b54bc92b2f0" containerName="authorino" Oct 30 01:46:38.944707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.944708 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="540c9d60-38b6-420b-aa85-402130d10f9c" containerName="authorino" Oct 30 01:46:38.949477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.949454 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:38.962411 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.962393 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 01:46:38.972758 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.972728 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-bwbbx\"" Oct 30 01:46:38.972911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.972856 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 01:46:38.974363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:38.974341 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pd2tp"] Oct 30 01:46:39.023045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.023017 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pd2tp\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.023186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.023050 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtnt\" (UniqueName: \"kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt\") pod \"ai-gateway-controller-7d85675f98-pd2tp\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.123723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.123691 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pd2tp\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.123723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.123728 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtnt\" (UniqueName: \"kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt\") pod \"ai-gateway-controller-7d85675f98-pd2tp\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.126185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.126165 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pd2tp\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.145186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.145158 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtnt\" (UniqueName: \"kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt\") pod \"ai-gateway-controller-7d85675f98-pd2tp\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.262160 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.262115 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:39.554716 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.554638 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pd2tp"] Oct 30 01:46:39.559293 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:39.559255 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3560d96d_a024_4a56_8113_fe6abbcf9736.slice/crio-626c87415b89e8f7e9fd1ac646c05ab7f0eb958e725d8b04cbde54429383f7e3 WatchSource:0}: Error finding container 626c87415b89e8f7e9fd1ac646c05ab7f0eb958e725d8b04cbde54429383f7e3: Status 404 returned error can't find the container with id 626c87415b89e8f7e9fd1ac646c05ab7f0eb958e725d8b04cbde54429383f7e3 Oct 30 01:46:39.829155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:39.829054 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" event={"ID":"3560d96d-a024-4a56-8113-fe6abbcf9736","Type":"ContainerStarted","Data":"626c87415b89e8f7e9fd1ac646c05ab7f0eb958e725d8b04cbde54429383f7e3"} Oct 30 01:46:41.841543 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:41.841505 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" event={"ID":"3560d96d-a024-4a56-8113-fe6abbcf9736","Type":"ContainerStarted","Data":"7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0"} Oct 30 01:46:41.842013 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:41.841634 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:41.880042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:41.879988 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" podStartSLOduration=2.346224204 podStartE2EDuration="3.879975489s" totalImagesPullingTime="1.533751285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:41.879334769 +0000 UTC m=+1030.271142613" watchObservedRunningTime="2026-10-30 01:46:41.879975489 +0000 UTC m=+1030.271783101" Oct 30 01:46:43.275189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.275155 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-qfc9q"] Oct 30 01:46:43.278674 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.278655 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.293792 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.293774 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 01:46:43.303314 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.303292 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jrbb4\"" Oct 30 01:46:43.304533 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.304509 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-qfc9q"] Oct 30 01:46:43.328451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.328428 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx"] Oct 30 01:46:43.332054 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.332035 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.352246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.352217 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx"] Oct 30 01:46:43.360512 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.360490 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume\") pod \"payload-processing-5f78687d5b-qfc9q\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.360639 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.360602 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfx42\" (UniqueName: \"kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42\") pod \"payload-processing-5f78687d5b-qfc9q\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.461482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.461449 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xsvkx\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.461675 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.461502 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfx42\" (UniqueName: \"kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42\") pod \"payload-processing-5f78687d5b-qfc9q\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.461675 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.461610 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xh4\" (UniqueName: \"kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4\") pod \"payload-pre-processing-7b98bb678c-xsvkx\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.461755 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.461702 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume\") pod \"payload-processing-5f78687d5b-qfc9q\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.462419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.462397 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume\") pod \"payload-processing-5f78687d5b-qfc9q\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.484793 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.484767 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfx42\" (UniqueName: \"kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42\") pod \"payload-processing-5f78687d5b-qfc9q\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.563117 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.563002 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xh4\" (UniqueName: \"kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4\") pod \"payload-pre-processing-7b98bb678c-xsvkx\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.563298 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.563143 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xsvkx\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.563856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.563831 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xsvkx\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.585112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.585079 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xh4\" (UniqueName: \"kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4\") pod \"payload-pre-processing-7b98bb678c-xsvkx\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.590894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.590860 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:43.644443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.644407 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:43.886793 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.886702 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-qfc9q"] Oct 30 01:46:43.890672 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:43.890636 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75c1729_dbe8_4486_a74d_fa6611610662.slice/crio-f70114b3e62b18859139db173759519eba73123ebd3a8e1680022b96322f2029 WatchSource:0}: Error finding container f70114b3e62b18859139db173759519eba73123ebd3a8e1680022b96322f2029: Status 404 returned error can't find the container with id f70114b3e62b18859139db173759519eba73123ebd3a8e1680022b96322f2029 Oct 30 01:46:43.946558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:43.946528 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx"] Oct 30 01:46:43.950978 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:43.950953 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77bd722_7979_4f70_a35b_e131590ff267.slice/crio-c24ab42c0d03f0f99b0500969c3c4701c20968a5309ac8bb1c5e2ac61f13d9ae WatchSource:0}: Error finding container c24ab42c0d03f0f99b0500969c3c4701c20968a5309ac8bb1c5e2ac61f13d9ae: Status 404 returned error can't find the container with id c24ab42c0d03f0f99b0500969c3c4701c20968a5309ac8bb1c5e2ac61f13d9ae Oct 30 01:46:44.858513 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:44.858476 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" event={"ID":"e77bd722-7979-4f70-a35b-e131590ff267","Type":"ContainerStarted","Data":"c24ab42c0d03f0f99b0500969c3c4701c20968a5309ac8bb1c5e2ac61f13d9ae"} Oct 30 01:46:44.860163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:44.860106 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" event={"ID":"e75c1729-dbe8-4486-a74d-fa6611610662","Type":"ContainerStarted","Data":"f70114b3e62b18859139db173759519eba73123ebd3a8e1680022b96322f2029"} Oct 30 01:46:46.695870 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:46:46.695832 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:46:46.870479 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:46.870439 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" event={"ID":"e77bd722-7979-4f70-a35b-e131590ff267","Type":"ContainerStarted","Data":"fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228"} Oct 30 01:46:46.870656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:46.870531 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:46:46.871924 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:46.871897 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" event={"ID":"e75c1729-dbe8-4486-a74d-fa6611610662","Type":"ContainerStarted","Data":"026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c"} Oct 30 01:46:46.872040 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:46.871949 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:46.908190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:46.908141 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" podStartSLOduration=1.645881904 podStartE2EDuration="3.908112906s" totalImagesPullingTime="2.262231002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:46.907252501 +0000 UTC m=+1035.299060134" watchObservedRunningTime="2026-10-30 01:46:46.908112906 +0000 UTC m=+1035.299920538" Oct 30 01:46:46.944135 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:46.944074 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" podStartSLOduration=1.582823303 podStartE2EDuration="3.944060295s" totalImagesPullingTime="2.361236992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:46.943511444 +0000 UTC m=+1035.335319077" watchObservedRunningTime="2026-10-30 01:46:46.944060295 +0000 UTC m=+1035.335867924" Oct 30 01:46:52.848576 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:52.848542 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:46:53.121507 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.121391 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-67f4f4fc68-mtts8"] Oct 30 01:46:53.124944 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.124923 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.146150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.146108 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67f4f4fc68-mtts8"] Oct 30 01:46:53.250210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.250173 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvts\" (UniqueName: \"kubernetes.io/projected/01787de0-9afd-46c6-b655-2aa41bd5f89e-kube-api-access-dgvts\") pod \"ai-gateway-controller-67f4f4fc68-mtts8\" (UID: \"01787de0-9afd-46c6-b655-2aa41bd5f89e\") " pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.250353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.250233 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01787de0-9afd-46c6-b655-2aa41bd5f89e-metrics-certs\") pod \"ai-gateway-controller-67f4f4fc68-mtts8\" (UID: \"01787de0-9afd-46c6-b655-2aa41bd5f89e\") " pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.350708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.350670 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01787de0-9afd-46c6-b655-2aa41bd5f89e-metrics-certs\") pod \"ai-gateway-controller-67f4f4fc68-mtts8\" (UID: \"01787de0-9afd-46c6-b655-2aa41bd5f89e\") " pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.350873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.350744 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvts\" (UniqueName: \"kubernetes.io/projected/01787de0-9afd-46c6-b655-2aa41bd5f89e-kube-api-access-dgvts\") pod \"ai-gateway-controller-67f4f4fc68-mtts8\" (UID: \"01787de0-9afd-46c6-b655-2aa41bd5f89e\") " pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.353270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.353234 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01787de0-9afd-46c6-b655-2aa41bd5f89e-metrics-certs\") pod \"ai-gateway-controller-67f4f4fc68-mtts8\" (UID: \"01787de0-9afd-46c6-b655-2aa41bd5f89e\") " pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.372564 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.372501 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvts\" (UniqueName: \"kubernetes.io/projected/01787de0-9afd-46c6-b655-2aa41bd5f89e-kube-api-access-dgvts\") pod \"ai-gateway-controller-67f4f4fc68-mtts8\" (UID: \"01787de0-9afd-46c6-b655-2aa41bd5f89e\") " pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.437016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.436987 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:53.736776 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:46:53.736748 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01787de0_9afd_46c6_b655_2aa41bd5f89e.slice/crio-5f86ed646b3c68ae37b5ead85d0dce3120a7ce8f00f56420bee447aeddf2132d WatchSource:0}: Error finding container 5f86ed646b3c68ae37b5ead85d0dce3120a7ce8f00f56420bee447aeddf2132d: Status 404 returned error can't find the container with id 5f86ed646b3c68ae37b5ead85d0dce3120a7ce8f00f56420bee447aeddf2132d Oct 30 01:46:53.737495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.737473 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67f4f4fc68-mtts8"] Oct 30 01:46:53.907797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:53.907761 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" event={"ID":"01787de0-9afd-46c6-b655-2aa41bd5f89e","Type":"ContainerStarted","Data":"5f86ed646b3c68ae37b5ead85d0dce3120a7ce8f00f56420bee447aeddf2132d"} Oct 30 01:46:54.913498 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:54.913465 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" event={"ID":"01787de0-9afd-46c6-b655-2aa41bd5f89e","Type":"ContainerStarted","Data":"b3991f364e16692ef75a01c77d50060cab02409ade62fc68fa977dbe17aac065"} Oct 30 01:46:54.913877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:54.913596 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:46:54.952085 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:54.952032 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" podStartSLOduration=1.6261320110000002 podStartE2EDuration="1.952019902s" totalImagesPullingTime="325.887891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:54.950933463 +0000 UTC m=+1043.342741095" watchObservedRunningTime="2026-10-30 01:46:54.952019902 +0000 UTC m=+1043.343827568" Oct 30 01:46:57.879358 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:57.879323 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 01:46:57.879859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:46:57.879399 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 01:47:05.920707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:05.920675 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-67f4f4fc68-mtts8" Oct 30 01:47:05.989800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:05.989764 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pd2tp"] Oct 30 01:47:05.990023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:05.989996 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" podUID="3560d96d-a024-4a56-8113-fe6abbcf9736" containerName="manager" containerID="cri-o://7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0" gracePeriod=30 Oct 30 01:47:06.228655 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.228632 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:47:06.260692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.260659 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs\") pod \"3560d96d-a024-4a56-8113-fe6abbcf9736\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " Oct 30 01:47:06.260838 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.260768 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt\" (UniqueName: \"kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt\") pod \"3560d96d-a024-4a56-8113-fe6abbcf9736\" (UID: \"3560d96d-a024-4a56-8113-fe6abbcf9736\") " Oct 30 01:47:06.262737 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.262702 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt" pod "3560d96d-a024-4a56-8113-fe6abbcf9736" (UID: "3560d96d-a024-4a56-8113-fe6abbcf9736"). InnerVolumeSpecName "kube-api-access-tvtnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:06.262861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.262711 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs" pod "3560d96d-a024-4a56-8113-fe6abbcf9736" (UID: "3560d96d-a024-4a56-8113-fe6abbcf9736"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:47:06.361552 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.361520 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3560d96d-a024-4a56-8113-fe6abbcf9736-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.361552 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.361547 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtnt\" (UniqueName: \"kubernetes.io/projected/3560d96d-a024-4a56-8113-fe6abbcf9736-kube-api-access-tvtnt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.971449 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.971414 6275 generic.go:356] "Generic (PLEG): container finished" podID="3560d96d-a024-4a56-8113-fe6abbcf9736" containerID="7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0" exitCode=0 Oct 30 01:47:06.971861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.971467 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" Oct 30 01:47:06.971861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.971502 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" event={"ID":"3560d96d-a024-4a56-8113-fe6abbcf9736","Type":"ContainerDied","Data":"7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0"} Oct 30 01:47:06.971861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.971561 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pd2tp" event={"ID":"3560d96d-a024-4a56-8113-fe6abbcf9736","Type":"ContainerDied","Data":"626c87415b89e8f7e9fd1ac646c05ab7f0eb958e725d8b04cbde54429383f7e3"} Oct 30 01:47:06.971861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.971587 6275 scope.go:122] "RemoveContainer" containerID="7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0" Oct 30 01:47:06.979994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.979978 6275 scope.go:122] "RemoveContainer" containerID="7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0" Oct 30 01:47:06.980285 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:47:06.980263 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0\": container with ID starting with 7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0 not found: ID does not exist" containerID="7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0" Oct 30 01:47:06.980382 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:06.980290 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0"} err="failed to get container status \"7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0\": rpc error: code = NotFound desc = could not find container \"7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0\": container with ID starting with 7c5eb8ee26c61ef64bb8d073a863f5c7b6ac5f35f40054f730bcdb8ceb5e7cf0 not found: ID does not exist" Oct 30 01:47:07.017833 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:07.017799 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pd2tp"] Oct 30 01:47:07.028366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:07.028346 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pd2tp"] Oct 30 01:47:07.736749 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:07.736715 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3560d96d-a024-4a56-8113-fe6abbcf9736" path="/var/lib/kubelet/pods/3560d96d-a024-4a56-8113-fe6abbcf9736/volumes" Oct 30 01:47:13.198169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.198115 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl"] Oct 30 01:47:13.198670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.198655 6275 cpu_manager.go:414] "containerMap: removing container" podUID="3560d96d-a024-4a56-8113-fe6abbcf9736" containerName="manager" Oct 30 01:47:13.198736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.198674 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3560d96d-a024-4a56-8113-fe6abbcf9736" containerName="manager" Oct 30 01:47:13.198786 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.198754 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3560d96d-a024-4a56-8113-fe6abbcf9736" containerName="manager" Oct 30 01:47:13.205181 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.205157 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.228896 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.228875 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9bctt\"" Oct 30 01:47:13.229070 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.228890 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 01:47:13.229070 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.228918 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 01:47:13.229281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.228925 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 01:47:13.230342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.230321 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl"] Oct 30 01:47:13.303509 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.303475 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.303509 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.303513 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3471b390-0a41-44c3-813c-61af6545a43c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.303692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.303535 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2gs\" (UniqueName: \"kubernetes.io/projected/3471b390-0a41-44c3-813c-61af6545a43c-kube-api-access-dp2gs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.303692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.303640 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.303764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.303695 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.303764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.303720 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.404631 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.404589 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.404811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.404671 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.404811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.404698 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.404811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.404745 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.404811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.404774 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3471b390-0a41-44c3-813c-61af6545a43c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.404811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.404804 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2gs\" (UniqueName: \"kubernetes.io/projected/3471b390-0a41-44c3-813c-61af6545a43c-kube-api-access-dp2gs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.405061 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.405033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.405148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.405070 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.405148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.405088 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.407082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.407059 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3471b390-0a41-44c3-813c-61af6545a43c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.407347 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.407330 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3471b390-0a41-44c3-813c-61af6545a43c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.428635 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.428603 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2gs\" (UniqueName: \"kubernetes.io/projected/3471b390-0a41-44c3-813c-61af6545a43c-kube-api-access-dp2gs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-srspl\" (UID: \"3471b390-0a41-44c3-813c-61af6545a43c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.518718 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.518685 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:13.816644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:13.816558 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl"] Oct 30 01:47:13.820450 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:13.820423 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3471b390_0a41_44c3_813c_61af6545a43c.slice/crio-8487cb71db3c5e7702acee7cb75b64d56d3bd1552acd2f378de61b1fe406a9e9 WatchSource:0}: Error finding container 8487cb71db3c5e7702acee7cb75b64d56d3bd1552acd2f378de61b1fe406a9e9: Status 404 returned error can't find the container with id 8487cb71db3c5e7702acee7cb75b64d56d3bd1552acd2f378de61b1fe406a9e9 Oct 30 01:47:14.003874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:14.003833 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" event={"ID":"3471b390-0a41-44c3-813c-61af6545a43c","Type":"ContainerStarted","Data":"8487cb71db3c5e7702acee7cb75b64d56d3bd1552acd2f378de61b1fe406a9e9"} Oct 30 01:47:15.679347 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.679315 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx"] Oct 30 01:47:15.683018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.682999 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.708515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.708485 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 01:47:15.710067 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.710039 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx"] Oct 30 01:47:15.724660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.724633 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.724789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.724670 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7vq\" (UniqueName: \"kubernetes.io/projected/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-kube-api-access-cg7vq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.724789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.724692 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.724872 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.724813 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.724912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.724877 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.724972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.724948 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825369 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825339 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825369 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825371 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825417 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825439 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7vq\" (UniqueName: \"kubernetes.io/projected/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-kube-api-access-cg7vq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825456 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825481 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825769 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.825909 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825875 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.826031 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.825991 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.827746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.827721 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.827999 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.827982 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.846642 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.846620 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7vq\" (UniqueName: \"kubernetes.io/projected/7bce4f69-7fd4-429a-9420-c6bcfbb52e04-kube-api-access-cg7vq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8mxxx\" (UID: \"7bce4f69-7fd4-429a-9420-c6bcfbb52e04\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:15.996679 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:15.996589 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:16.015675 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:16.015634 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" event={"ID":"3471b390-0a41-44c3-813c-61af6545a43c","Type":"ContainerStarted","Data":"13bc54fc461767dfa07e1f50654632fe8632f059b897fc03145dc7b57757b5e5"} Oct 30 01:47:16.057148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:16.057058 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" podStartSLOduration=1.7982840329999998 podStartE2EDuration="3.057039114s" totalImagesPullingTime="1.258755081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:16.056165108 +0000 UTC m=+1064.447972742" watchObservedRunningTime="2026-10-30 01:47:16.057039114 +0000 UTC m=+1064.448846748" Oct 30 01:47:16.293881 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:16.293843 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx"] Oct 30 01:47:16.298094 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:16.298067 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bce4f69_7fd4_429a_9420_c6bcfbb52e04.slice/crio-aa04bcedda5ba6c189e6c0ad0c9a132368f6ad8f2a6179b10306e8ed57eb43e0 WatchSource:0}: Error finding container aa04bcedda5ba6c189e6c0ad0c9a132368f6ad8f2a6179b10306e8ed57eb43e0: Status 404 returned error can't find the container with id aa04bcedda5ba6c189e6c0ad0c9a132368f6ad8f2a6179b10306e8ed57eb43e0 Oct 30 01:47:16.519808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:16.519787 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:16.695533 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:47:16.695499 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:47:17.021873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:17.021834 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" event={"ID":"7bce4f69-7fd4-429a-9420-c6bcfbb52e04","Type":"ContainerStarted","Data":"8d64c04f40088f75ad5384f3ae555cbb5b20869243e4c5105b217feb2a7d6007"} Oct 30 01:47:17.022039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:17.021899 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" event={"ID":"7bce4f69-7fd4-429a-9420-c6bcfbb52e04","Type":"ContainerStarted","Data":"aa04bcedda5ba6c189e6c0ad0c9a132368f6ad8f2a6179b10306e8ed57eb43e0"} Oct 30 01:47:17.064288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:17.064236 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" podStartSLOduration=1.8649919499999998 podStartE2EDuration="2.064217087s" totalImagesPullingTime="199.225137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:17.062363581 +0000 UTC m=+1065.454171218" watchObservedRunningTime="2026-10-30 01:47:17.064217087 +0000 UTC m=+1065.456024720" Oct 30 01:47:17.996767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:17.996733 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:21.104886 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.104849 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6"] Oct 30 01:47:21.108900 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.108878 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.123253 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.123224 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 01:47:21.135744 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.135710 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6"] Oct 30 01:47:21.172484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.172444 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.172669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.172511 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.172669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.172593 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.172669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.172625 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e7d3d4-3f98-40c5-9a45-17237ebb244a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.172669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.172652 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9hl\" (UniqueName: \"kubernetes.io/projected/56e7d3d4-3f98-40c5-9a45-17237ebb244a-kube-api-access-4q9hl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.172900 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.172796 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.273904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.273864 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.273904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.273902 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e7d3d4-3f98-40c5-9a45-17237ebb244a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.273923 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9hl\" (UniqueName: \"kubernetes.io/projected/56e7d3d4-3f98-40c5-9a45-17237ebb244a-kube-api-access-4q9hl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.274051 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.274168 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.274208 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274468 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.274437 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274614 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.274524 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.274614 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.274597 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.276324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.276303 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e7d3d4-3f98-40c5-9a45-17237ebb244a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.276721 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.276696 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e7d3d4-3f98-40c5-9a45-17237ebb244a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.295462 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.295437 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9hl\" (UniqueName: \"kubernetes.io/projected/56e7d3d4-3f98-40c5-9a45-17237ebb244a-kube-api-access-4q9hl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6\" (UID: \"56e7d3d4-3f98-40c5-9a45-17237ebb244a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.426665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.426573 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:21.742370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:21.742276 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6"] Oct 30 01:47:21.746250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:21.746222 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e7d3d4_3f98_40c5_9a45_17237ebb244a.slice/crio-2e61cc4329f02c6c9797a71709ee9ed8f5ffab93487692a34d6f19ad93d1ca6e WatchSource:0}: Error finding container 2e61cc4329f02c6c9797a71709ee9ed8f5ffab93487692a34d6f19ad93d1ca6e: Status 404 returned error can't find the container with id 2e61cc4329f02c6c9797a71709ee9ed8f5ffab93487692a34d6f19ad93d1ca6e Oct 30 01:47:22.047418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:22.047386 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" event={"ID":"56e7d3d4-3f98-40c5-9a45-17237ebb244a","Type":"ContainerStarted","Data":"2f6634ef2a380152c89f1cead8bfaf6f88b23e176106bf48cc83884c3af2abec"} Oct 30 01:47:22.047602 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:22.047439 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" event={"ID":"56e7d3d4-3f98-40c5-9a45-17237ebb244a","Type":"ContainerStarted","Data":"2e61cc4329f02c6c9797a71709ee9ed8f5ffab93487692a34d6f19ad93d1ca6e"} Oct 30 01:47:22.087681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:22.087625 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" podStartSLOduration=0.89693445 podStartE2EDuration="1.087611924s" totalImagesPullingTime="190.677474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:22.086588507 +0000 UTC m=+1070.478396141" watchObservedRunningTime="2026-10-30 01:47:22.087611924 +0000 UTC m=+1070.479419557" Oct 30 01:47:22.426909 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:22.426834 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:23.527144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:23.525164 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:23.548139 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:23.547433 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:23.991023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:23.990929 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd"] Oct 30 01:47:23.995098 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:23.995075 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.008599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.008570 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 01:47:24.009906 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.009879 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.010095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.009918 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.010095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.009942 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.010095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.010033 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.010095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.010062 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30d76-8aee-4c30-bfe6-66c5a9766419-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.010095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.010090 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tvs\" (UniqueName: \"kubernetes.io/projected/cac30d76-8aee-4c30-bfe6-66c5a9766419-kube-api-access-78tvs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.021787 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.021762 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd"] Oct 30 01:47:24.069042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.069014 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-srspl" Oct 30 01:47:24.110659 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.110614 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30d76-8aee-4c30-bfe6-66c5a9766419-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.110821 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.110699 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tvs\" (UniqueName: \"kubernetes.io/projected/cac30d76-8aee-4c30-bfe6-66c5a9766419-kube-api-access-78tvs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.110821 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.110764 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.110821 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.110807 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.110997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.110836 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.110997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.110938 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.111487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.111372 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.111487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.111398 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.111487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.111448 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.113155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.113115 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30d76-8aee-4c30-bfe6-66c5a9766419-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.113227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.113197 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30d76-8aee-4c30-bfe6-66c5a9766419-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.132662 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.132637 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tvs\" (UniqueName: \"kubernetes.io/projected/cac30d76-8aee-4c30-bfe6-66c5a9766419-kube-api-access-78tvs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd\" (UID: \"cac30d76-8aee-4c30-bfe6-66c5a9766419\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.309890 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.309851 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:24.608791 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:24.608695 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd"] Oct 30 01:47:24.612683 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:24.612655 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac30d76_8aee_4c30_bfe6_66c5a9766419.slice/crio-f044947181d6797a0b3da14406a975699475fe4a49897684e3b324c77515ce75 WatchSource:0}: Error finding container f044947181d6797a0b3da14406a975699475fe4a49897684e3b324c77515ce75: Status 404 returned error can't find the container with id f044947181d6797a0b3da14406a975699475fe4a49897684e3b324c77515ce75 Oct 30 01:47:25.062921 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:25.062881 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" event={"ID":"cac30d76-8aee-4c30-bfe6-66c5a9766419","Type":"ContainerStarted","Data":"46936a25895db302eb76606d4397cf4cc3887ebb7d1911e865f7cfea94c73683"} Oct 30 01:47:25.063076 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:25.062971 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" event={"ID":"cac30d76-8aee-4c30-bfe6-66c5a9766419","Type":"ContainerStarted","Data":"f044947181d6797a0b3da14406a975699475fe4a49897684e3b324c77515ce75"} Oct 30 01:47:25.102781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:25.102728 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" podStartSLOduration=1.917416216 podStartE2EDuration="2.102714939s" totalImagesPullingTime="185.298723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:25.101696171 +0000 UTC m=+1073.493503817" watchObservedRunningTime="2026-10-30 01:47:25.102714939 +0000 UTC m=+1073.494522551" Oct 30 01:47:25.310523 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:25.310479 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:25.996990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:25.996954 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:26.010130 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:26.010100 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:26.083097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:26.083059 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8mxxx" Oct 30 01:47:31.427194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:31.427154 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:31.439958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:31.439931 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:32.154535 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:32.154448 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6" Oct 30 01:47:34.310730 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:34.310692 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:34.323491 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:34.323460 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:35.168971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:35.168936 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd" Oct 30 01:47:40.182562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.182526 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr"] Oct 30 01:47:40.187421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.187393 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.201582 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.201560 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5dmpm\"" Oct 30 01:47:40.216281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.216256 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr"] Oct 30 01:47:40.249852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.249815 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.249994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.249853 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.249994 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.249983 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.250092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.250010 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.250092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.250040 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.250092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.250069 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6xp\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.250291 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.250170 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.250291 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.250195 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.250415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.250319 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351443 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351481 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351515 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351551 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351578 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351653 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351686 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.351722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351716 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.352027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.351915 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6xp\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.352087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.352026 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.352164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.352103 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.352221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.352157 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.352313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.352295 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.352489 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.352473 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.354361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.354339 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.354689 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.354669 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.373392 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.373360 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.373660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.373633 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6xp\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp\") pod \"maas-default-gateway-openshift-default-75d7c785f-fl6gr\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.500517 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.500434 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:40.823359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:40.823268 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr"] Oct 30 01:47:40.826840 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:40.826811 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efcb4b3_4afb_4420_a3d2_67017dc0ba4c.slice/crio-8b0e3ca9f4a1f2e658a00e6b71749d5c0229ccdcf86a791bd3499caa9c3ace03 WatchSource:0}: Error finding container 8b0e3ca9f4a1f2e658a00e6b71749d5c0229ccdcf86a791bd3499caa9c3ace03: Status 404 returned error can't find the container with id 8b0e3ca9f4a1f2e658a00e6b71749d5c0229ccdcf86a791bd3499caa9c3ace03 Oct 30 01:47:41.185017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:41.184931 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" event={"ID":"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c","Type":"ContainerStarted","Data":"8b0e3ca9f4a1f2e658a00e6b71749d5c0229ccdcf86a791bd3499caa9c3ace03"} Oct 30 01:47:42.159626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.158245 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7795bf7868-gmhpg"] Oct 30 01:47:42.162046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.162022 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.181986 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.181923 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7795bf7868-gmhpg"] Oct 30 01:47:42.269334 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.269300 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3-tls-cert\") pod \"authorino-7795bf7868-gmhpg\" (UID: \"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3\") " pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.269782 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.269393 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8db\" (UniqueName: \"kubernetes.io/projected/09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3-kube-api-access-kg8db\") pod \"authorino-7795bf7868-gmhpg\" (UID: \"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3\") " pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.370403 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.370358 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3-tls-cert\") pod \"authorino-7795bf7868-gmhpg\" (UID: \"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3\") " pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.370600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.370461 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8db\" (UniqueName: \"kubernetes.io/projected/09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3-kube-api-access-kg8db\") pod \"authorino-7795bf7868-gmhpg\" (UID: \"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3\") " pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.372951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.372924 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3-tls-cert\") pod \"authorino-7795bf7868-gmhpg\" (UID: \"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3\") " pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.400884 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.400857 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8db\" (UniqueName: \"kubernetes.io/projected/09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3-kube-api-access-kg8db\") pod \"authorino-7795bf7868-gmhpg\" (UID: \"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3\") " pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.477481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.477401 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7795bf7868-gmhpg" Oct 30 01:47:42.785373 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:42.785336 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7795bf7868-gmhpg"] Oct 30 01:47:42.789159 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:42.789090 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09f2fd17_38f2_4f7d_9cec_c302c7bfdcb3.slice/crio-a36c268ab8e58fe233b3d83911e46e2d26765a1ace2a5d9e0a9e770b21ff080e WatchSource:0}: Error finding container a36c268ab8e58fe233b3d83911e46e2d26765a1ace2a5d9e0a9e770b21ff080e: Status 404 returned error can't find the container with id a36c268ab8e58fe233b3d83911e46e2d26765a1ace2a5d9e0a9e770b21ff080e Oct 30 01:47:43.196380 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:43.196289 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7795bf7868-gmhpg" event={"ID":"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3","Type":"ContainerStarted","Data":"a36c268ab8e58fe233b3d83911e46e2d26765a1ace2a5d9e0a9e770b21ff080e"} Oct 30 01:47:43.787660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:43.787617 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:47:43.787967 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:43.787706 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:47:43.787967 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:43.787751 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:47:44.203189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.202230 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7795bf7868-gmhpg" event={"ID":"09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3","Type":"ContainerStarted","Data":"e3719b537f73275ee255fe31fb944265acb30523d1d03354a317575b3238cc88"} Oct 30 01:47:44.204207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.204174 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" event={"ID":"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c","Type":"ContainerStarted","Data":"dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be"} Oct 30 01:47:44.241671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.241345 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7795bf7868-gmhpg" podStartSLOduration=1.652471198 podStartE2EDuration="2.241326998s" totalImagesPullingTime="588.8558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:44.240300129 +0000 UTC m=+1092.632107763" watchObservedRunningTime="2026-10-30 01:47:44.241326998 +0000 UTC m=+1092.633134631" Oct 30 01:47:44.274168 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.274110 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59cdc7947-244dk"] Oct 30 01:47:44.274673 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.274622 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-59cdc7947-244dk" podUID="6ef337d6-2b13-4ff8-8a55-333a6cdae799" containerName="authorino" containerID="cri-o://b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5" gracePeriod=30 Oct 30 01:47:44.288969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.286425 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podStartSLOduration=1.328643064 podStartE2EDuration="4.286407208s" totalImagesPullingTime="2.957764144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:44.285045121 +0000 UTC m=+1092.676852755" watchObservedRunningTime="2026-10-30 01:47:44.286407208 +0000 UTC m=+1092.678214842" Oct 30 01:47:44.509948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.503084 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:44.509948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.507402 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:44.509948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.507545 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:44.593009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.592983 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:47:44.710615 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.710582 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert\" (UniqueName: \"kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert\") pod \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " Oct 30 01:47:44.710846 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.710623 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs\" (UniqueName: \"kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs\") pod \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\" (UID: \"6ef337d6-2b13-4ff8-8a55-333a6cdae799\") " Oct 30 01:47:44.712741 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.712707 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs" pod "6ef337d6-2b13-4ff8-8a55-333a6cdae799" (UID: "6ef337d6-2b13-4ff8-8a55-333a6cdae799"). InnerVolumeSpecName "kube-api-access-9zdbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:44.720948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.720921 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert" pod "6ef337d6-2b13-4ff8-8a55-333a6cdae799" (UID: "6ef337d6-2b13-4ff8-8a55-333a6cdae799"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:47:44.811720 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.811684 6275 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ef337d6-2b13-4ff8-8a55-333a6cdae799-tls-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:47:44.811720 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:44.811716 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdbs\" (UniqueName: \"kubernetes.io/projected/6ef337d6-2b13-4ff8-8a55-333a6cdae799-kube-api-access-9zdbs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:47:45.210904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.210821 6275 generic.go:356] "Generic (PLEG): container finished" podID="6ef337d6-2b13-4ff8-8a55-333a6cdae799" containerID="b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5" exitCode=0 Oct 30 01:47:45.210904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.210867 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cdc7947-244dk" Oct 30 01:47:45.210904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.210895 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cdc7947-244dk" event={"ID":"6ef337d6-2b13-4ff8-8a55-333a6cdae799","Type":"ContainerDied","Data":"b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5"} Oct 30 01:47:45.211162 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.210940 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cdc7947-244dk" event={"ID":"6ef337d6-2b13-4ff8-8a55-333a6cdae799","Type":"ContainerDied","Data":"7245be81dd5dfbc8d37ff0eddcd4b863af09a18971621a10eef0e08be73b9aec"} Oct 30 01:47:45.211162 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.210962 6275 scope.go:122] "RemoveContainer" containerID="b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5" Oct 30 01:47:45.221834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.221815 6275 scope.go:122] "RemoveContainer" containerID="b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5" Oct 30 01:47:45.222229 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:47:45.222205 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5\": container with ID starting with b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5 not found: ID does not exist" containerID="b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5" Oct 30 01:47:45.222425 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.222237 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5"} err="failed to get container status \"b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5\": rpc error: code = NotFound desc = could not find container \"b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5\": container with ID starting with b816ca87ae5870df31746047593d449cae1d059bdecc2469bd77534b35169fb5 not found: ID does not exist" Oct 30 01:47:45.280692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.280630 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59cdc7947-244dk"] Oct 30 01:47:45.284110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.284082 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59cdc7947-244dk"] Oct 30 01:47:45.501710 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.501627 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:45.501710 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.501698 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:45.737100 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:45.737063 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef337d6-2b13-4ff8-8a55-333a6cdae799" path="/var/lib/kubelet/pods/6ef337d6-2b13-4ff8-8a55-333a6cdae799/volumes" Oct 30 01:47:46.501194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:46.501153 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:46.501665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:46.501231 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:46.699573 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:47:46.699533 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:47:47.501158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:47.501096 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:47.501328 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:47.501188 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:48.501645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:48.501601 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:48.502026 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:48.501678 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:49.501422 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:49.501383 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:49.501612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:49.501469 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:50.501767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:50.501729 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:50.502193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:50.501799 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:51.501530 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:51.501492 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:47:51.501706 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:51.501559 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 01:47:52.505209 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:52.505166 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:52.505621 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:52.505600 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:52.506261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:52.506242 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 01:47:55.244737 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.244700 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g"] Oct 30 01:47:55.245137 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.245094 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6ef337d6-2b13-4ff8-8a55-333a6cdae799" containerName="authorino" Oct 30 01:47:55.245137 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.245106 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef337d6-2b13-4ff8-8a55-333a6cdae799" containerName="authorino" Oct 30 01:47:55.245235 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.245214 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef337d6-2b13-4ff8-8a55-333a6cdae799" containerName="authorino" Oct 30 01:47:55.250445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.250425 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.275935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.275794 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g"] Oct 30 01:47:55.345681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345644 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.345681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345686 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdpc\" (UniqueName: \"kubernetes.io/projected/df283d5f-6e54-406a-b11a-9da1af27074b-kube-api-access-svdpc\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.345902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345709 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df283d5f-6e54-406a-b11a-9da1af27074b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.345902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345828 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.345902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345874 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.345902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345892 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.346025 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345954 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df283d5f-6e54-406a-b11a-9da1af27074b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.346025 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.345973 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df283d5f-6e54-406a-b11a-9da1af27074b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.346025 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.346011 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.446818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.446780 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.446997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.446822 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.446997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.446870 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df283d5f-6e54-406a-b11a-9da1af27074b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.446997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.446892 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df283d5f-6e54-406a-b11a-9da1af27074b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.446997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.446965 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447032 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447076 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdpc\" (UniqueName: \"kubernetes.io/projected/df283d5f-6e54-406a-b11a-9da1af27074b-kube-api-access-svdpc\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447105 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df283d5f-6e54-406a-b11a-9da1af27074b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447199 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447368 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447450 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447574 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447675 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447658 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.447969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.447947 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df283d5f-6e54-406a-b11a-9da1af27074b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.449290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.449266 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df283d5f-6e54-406a-b11a-9da1af27074b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.449540 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.449520 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df283d5f-6e54-406a-b11a-9da1af27074b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.468912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.468887 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df283d5f-6e54-406a-b11a-9da1af27074b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.469101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.469078 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdpc\" (UniqueName: \"kubernetes.io/projected/df283d5f-6e54-406a-b11a-9da1af27074b-kube-api-access-svdpc\") pod \"maas-default-gateway-openshift-default-75d7c785f-nzg5g\" (UID: \"df283d5f-6e54-406a-b11a-9da1af27074b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.565268 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.565236 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:55.878360 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.878273 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g"] Oct 30 01:47:55.882091 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:47:55.882065 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf283d5f_6e54_406a_b11a_9da1af27074b.slice/crio-e45ad0a7b988b5ae846ef201ea135d917cd768211e78a50511f285175c89dc66 WatchSource:0}: Error finding container e45ad0a7b988b5ae846ef201ea135d917cd768211e78a50511f285175c89dc66: Status 404 returned error can't find the container with id e45ad0a7b988b5ae846ef201ea135d917cd768211e78a50511f285175c89dc66 Oct 30 01:47:55.885206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.885172 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:47:55.885282 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.885240 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:47:55.885282 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:55.885271 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:47:56.283697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:56.282645 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" event={"ID":"df283d5f-6e54-406a-b11a-9da1af27074b","Type":"ContainerStarted","Data":"e0285f2b13b7273dede2b9d0c80862e608b8ba6517f8388e33b81d0ba77f13f8"} Oct 30 01:47:56.283697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:56.282717 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" event={"ID":"df283d5f-6e54-406a-b11a-9da1af27074b","Type":"ContainerStarted","Data":"e45ad0a7b988b5ae846ef201ea135d917cd768211e78a50511f285175c89dc66"} Oct 30 01:47:56.330630 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:56.330578 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podStartSLOduration=1.33056362 podStartE2EDuration="1.33056362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:56.329847124 +0000 UTC m=+1104.721654757" watchObservedRunningTime="2026-10-30 01:47:56.33056362 +0000 UTC m=+1104.722371234" Oct 30 01:47:56.565450 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:56.565355 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:47:56.566884 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:56.566853 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:56.567015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:56.566904 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:47:57.566306 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:57.566272 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:57.566708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:57.566325 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:47:58.566203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:58.566164 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:58.566364 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:58.566221 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:47:59.566060 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:59.566014 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:59.566264 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:47:59.566109 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:48:00.565771 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:00.565728 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:48:00.566271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:00.565796 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:48:01.566250 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:01.566216 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:48:01.566656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:01.566282 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:48:02.566462 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:02.566424 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:48:02.566965 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:02.566505 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 01:48:04.049156 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:04.049105 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:48:04.049626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:04.049511 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:48:04.050252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:04.050229 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" Oct 30 01:48:15.276008 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.275970 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r"] Oct 30 01:48:15.284740 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.284711 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.311701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.311666 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-61b6469a-openshift-default-dockercfg-f958z\"" Oct 30 01:48:15.319369 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.319340 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r"] Oct 30 01:48:15.330175 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330145 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330213 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330297 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330326 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330448 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330550 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330488 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330550 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330523 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330659 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330566 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgts\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.330659 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.330650 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432107 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432069 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432107 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432115 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432148 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432165 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgts\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432194 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432268 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432325 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432358 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432384 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432577 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432625 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432885 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432862 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.432997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.432978 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.433206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.433180 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.434801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.434779 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.435003 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.434980 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.464663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.464632 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgts\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.464663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.464644 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token\") pod \"e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.603317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.603231 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:15.940770 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.940678 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r"] Oct 30 01:48:15.944763 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:15.944732 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d45657_27b5_4070_aadc_b93e94487605.slice/crio-b023124921b36acbdb8d47c2cc4987d4cdff07e6e61eb896c2043b31062cb73b WatchSource:0}: Error finding container b023124921b36acbdb8d47c2cc4987d4cdff07e6e61eb896c2043b31062cb73b: Status 404 returned error can't find the container with id b023124921b36acbdb8d47c2cc4987d4cdff07e6e61eb896c2043b31062cb73b Oct 30 01:48:15.947191 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.947148 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:15.947294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.947241 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:15.947294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:15.947286 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:16.381564 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.381529 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" event={"ID":"67d45657-27b5-4070-aadc-b93e94487605","Type":"ContainerStarted","Data":"cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c"} Oct 30 01:48:16.381914 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.381587 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" event={"ID":"67d45657-27b5-4070-aadc-b93e94487605","Type":"ContainerStarted","Data":"b023124921b36acbdb8d47c2cc4987d4cdff07e6e61eb896c2043b31062cb73b"} Oct 30 01:48:16.424868 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.424813 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podStartSLOduration=1.424797769 podStartE2EDuration="1.424797769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:16.423618399 +0000 UTC m=+1124.815426032" watchObservedRunningTime="2026-10-30 01:48:16.424797769 +0000 UTC m=+1124.816605402" Oct 30 01:48:16.603959 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.603927 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:16.608840 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.608814 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:16.696068 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:16.695990 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:48:16.960254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.959683 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh"] Oct 30 01:48:16.964808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.964783 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:16.978435 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.978409 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-14b53b75-openshift-default-dockercfg-hhmff\"" Oct 30 01:48:16.990980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:16.990951 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh"] Oct 30 01:48:17.055459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.055418 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.055808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.055783 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056038 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056002 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056046 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056078 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056146 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056174 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056197 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpfg\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.056665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.056284 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.156974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.156942 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.156974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.156986 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157004 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157052 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157086 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157152 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157174 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157198 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpfg\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157541 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157277 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157541 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157412 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157541 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157472 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157597 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157633 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.157940 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.157913 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.159658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.159637 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.159908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.159891 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.180034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.180008 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.180165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.180071 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpfg\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg\") pod \"e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.280922 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.280878 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:17.386724 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.386693 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:17.387816 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.387793 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:17.599297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.599201 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh"] Oct 30 01:48:17.602260 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:17.602232 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7404d52a_793e_4bd1_a262_8ee7e49e0c8e.slice/crio-2b44edd38faed21804f2069c7524414d6acf4b158f3f09be3c0810983da75f67 WatchSource:0}: Error finding container 2b44edd38faed21804f2069c7524414d6acf4b158f3f09be3c0810983da75f67: Status 404 returned error can't find the container with id 2b44edd38faed21804f2069c7524414d6acf4b158f3f09be3c0810983da75f67 Oct 30 01:48:17.604911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.604877 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:17.604996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.604953 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:17.605059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:17.604997 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:18.392979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:18.392936 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" event={"ID":"7404d52a-793e-4bd1-a262-8ee7e49e0c8e","Type":"ContainerStarted","Data":"f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8"} Oct 30 01:48:18.393401 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:18.393015 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" event={"ID":"7404d52a-793e-4bd1-a262-8ee7e49e0c8e","Type":"ContainerStarted","Data":"2b44edd38faed21804f2069c7524414d6acf4b158f3f09be3c0810983da75f67"} Oct 30 01:48:19.281723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.281682 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:19.287042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.287014 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:19.340188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.340095 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podStartSLOduration=3.340081213 podStartE2EDuration="3.340081213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:18.45052752 +0000 UTC m=+1126.842335157" watchObservedRunningTime="2026-10-30 01:48:19.340081213 +0000 UTC m=+1127.731888847" Oct 30 01:48:19.398675 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.398647 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:19.399652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.399629 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:19.798681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.798649 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s"] Oct 30 01:48:19.802044 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.802022 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:19.816514 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.816491 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bc6b04f0-dockercfg-b4wfr\"" Oct 30 01:48:19.816514 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.816508 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bc6b04f0\"" Oct 30 01:48:19.823094 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.823067 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s"] Oct 30 01:48:19.890954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.887298 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hs7q\" (UniqueName: \"kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q\") pod \"payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:19.890954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.887492 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:19.890954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.887629 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw"] Oct 30 01:48:19.892672 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.892644 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:19.909760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.909725 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw"] Oct 30 01:48:19.988487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.988451 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hs7q\" (UniqueName: \"kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q\") pod \"payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:19.988655 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.988561 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgpbs\" (UniqueName: \"kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs\") pod \"payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:19.988655 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.988602 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:19.988655 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.988619 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:19.989301 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:19.989283 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:20.010273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.010238 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hs7q\" (UniqueName: \"kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q\") pod \"payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:20.089172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.089052 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpbs\" (UniqueName: \"kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs\") pod \"payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:20.089172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.089097 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:20.089800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.089777 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:20.111497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.111462 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpbs\" (UniqueName: \"kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs\") pod \"payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:20.115172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.115112 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:20.208792 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.208764 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:20.413275 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.413188 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s"] Oct 30 01:48:20.417777 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:20.417750 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2c120d_59cc_4e41_9105_44e64f37893d.slice/crio-c14015c06623a8481c8f26b9fa1f9f3432f6c84847550a0f477ba0051a2cf582 WatchSource:0}: Error finding container c14015c06623a8481c8f26b9fa1f9f3432f6c84847550a0f477ba0051a2cf582: Status 404 returned error can't find the container with id c14015c06623a8481c8f26b9fa1f9f3432f6c84847550a0f477ba0051a2cf582 Oct 30 01:48:20.505654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.505585 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw"] Oct 30 01:48:20.509036 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:20.509013 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f69095_f427_4566_ae48_72cf0af181c8.slice/crio-69eeef149024e605686e976f8cac39baedab9b30594dc4348d30adfe2655a0d3 WatchSource:0}: Error finding container 69eeef149024e605686e976f8cac39baedab9b30594dc4348d30adfe2655a0d3: Status 404 returned error can't find the container with id 69eeef149024e605686e976f8cac39baedab9b30594dc4348d30adfe2655a0d3 Oct 30 01:48:20.858269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.858235 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q"] Oct 30 01:48:20.862086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.862063 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.889653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.889620 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q"] Oct 30 01:48:20.896114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.896020 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.896114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.896070 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.896362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.896302 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.896362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.896345 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpf8\" (UniqueName: \"kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.914096 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.914068 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bc6b04f0-serving-cert\"" Oct 30 01:48:20.996891 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.996865 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.997027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.996900 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpf8\" (UniqueName: \"kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.997027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.996977 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.997027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.997011 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.997777 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.997751 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.999443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.999413 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:20.999557 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:20.999459 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:21.019302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.019276 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpf8\" (UniqueName: \"kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8\") pod \"maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:21.174587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.174511 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:21.409511 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.409480 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" event={"ID":"d2f69095-f427-4566-ae48-72cf0af181c8","Type":"ContainerStarted","Data":"b1c885fe3654535fa2c81febf464c237fcf0f0ddbe9239994f1f5f5ea5e51c00"} Oct 30 01:48:21.409658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.409555 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" event={"ID":"d2f69095-f427-4566-ae48-72cf0af181c8","Type":"ContainerStarted","Data":"69eeef149024e605686e976f8cac39baedab9b30594dc4348d30adfe2655a0d3"} Oct 30 01:48:21.409658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.409600 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:21.411304 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.411272 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" event={"ID":"ac2c120d-59cc-4e41-9105-44e64f37893d","Type":"ContainerStarted","Data":"a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3"} Oct 30 01:48:21.411425 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.411321 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" event={"ID":"ac2c120d-59cc-4e41-9105-44e64f37893d","Type":"ContainerStarted","Data":"c14015c06623a8481c8f26b9fa1f9f3432f6c84847550a0f477ba0051a2cf582"} Oct 30 01:48:21.411503 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.411459 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:21.457738 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.457653 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" podStartSLOduration=2.457640423 podStartE2EDuration="2.457640423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:21.455797853 +0000 UTC m=+1129.847605487" watchObservedRunningTime="2026-10-30 01:48:21.457640423 +0000 UTC m=+1129.849448084" Oct 30 01:48:21.495476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.495432 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" podStartSLOduration=2.495417783 podStartE2EDuration="2.495417783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:21.494185429 +0000 UTC m=+1129.885993062" watchObservedRunningTime="2026-10-30 01:48:21.495417783 +0000 UTC m=+1129.887225407" Oct 30 01:48:21.501733 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:21.501481 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q"] Oct 30 01:48:22.417697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.417661 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" event={"ID":"9b4f5c09-fbf8-4815-b8a2-62b0a000f512","Type":"ContainerStarted","Data":"c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4"} Oct 30 01:48:22.417895 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.417726 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" event={"ID":"9b4f5c09-fbf8-4815-b8a2-62b0a000f512","Type":"ContainerStarted","Data":"a5eb6c00f93076e1bfd85b441050ee57cc2ee07a44c955aec8a86485120156e8"} Oct 30 01:48:22.417973 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.417940 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:22.459635 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.459587 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" podStartSLOduration=2.032594487 podStartE2EDuration="2.459573966s" totalImagesPullingTime="426.979479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:22.458034214 +0000 UTC m=+1130.849841847" watchObservedRunningTime="2026-10-30 01:48:22.459573966 +0000 UTC m=+1130.851381610" Oct 30 01:48:22.589346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.589317 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr"] Oct 30 01:48:22.593068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.593048 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.611901 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.611860 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ns8f\" (UniqueName: \"kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f\") pod \"payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.612034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.611999 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.614542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.614519 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr"] Oct 30 01:48:22.619337 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.619306 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7664a237\"" Oct 30 01:48:22.619476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.619448 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7664a237-dockercfg-cvfrm\"" Oct 30 01:48:22.713323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.713241 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ns8f\" (UniqueName: \"kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f\") pod \"payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.713463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.713338 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.714010 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.713988 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.735261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.735227 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ns8f\" (UniqueName: \"kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f\") pod \"payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:22.906161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:22.906113 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:23.014910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.014879 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp"] Oct 30 01:48:23.019945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.019926 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.038516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.038493 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7664a237-serving-cert\"" Oct 30 01:48:23.045779 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.045749 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp"] Oct 30 01:48:23.117898 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.117863 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.118055 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.117921 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.118055 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.118016 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.118166 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.118076 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.205943 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.205020 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr"] Oct 30 01:48:23.207816 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:23.207783 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea416368_c50c_4926_aafb_b6ae7ec0c790.slice/crio-b17022635ad2eb31d05a97dd2afaabdea427fc53abba732db7bb9fbfd91de711 WatchSource:0}: Error finding container b17022635ad2eb31d05a97dd2afaabdea427fc53abba732db7bb9fbfd91de711: Status 404 returned error can't find the container with id b17022635ad2eb31d05a97dd2afaabdea427fc53abba732db7bb9fbfd91de711 Oct 30 01:48:23.219308 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.219286 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.219421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.219331 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.219421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.219376 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.219528 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.219422 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.219528 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:23.219517 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-serving-cert: secret "maas-api-e2e-migrate-7664a237-serving-cert" not found Oct 30 01:48:23.219622 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:23.219582 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls podName:f506cd24-b676-4179-84e3-4c8027bced34 nodeName:}" failed. No retries permitted until 2026-10-30 01:48:23.719565577 +0000 UTC m=+1132.111373188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls") pod "maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" (UID: "f506cd24-b676-4179-84e3-4c8027bced34") : secret "maas-api-e2e-migrate-7664a237-serving-cert" not found Oct 30 01:48:23.220082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.220065 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.221981 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.221961 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.241061 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.241034 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.424800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.424764 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" event={"ID":"ea416368-c50c-4926-aafb-b6ae7ec0c790","Type":"ContainerStarted","Data":"666bc3e9a558e95040ca2a4a91e5c60386dfaf82bed21dfbd9111c6197457773"} Oct 30 01:48:23.425013 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.424822 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" event={"ID":"ea416368-c50c-4926-aafb-b6ae7ec0c790","Type":"ContainerStarted","Data":"b17022635ad2eb31d05a97dd2afaabdea427fc53abba732db7bb9fbfd91de711"} Oct 30 01:48:23.425086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.425035 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:23.463928 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.463881 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" podStartSLOduration=1.463860006 podStartE2EDuration="1.463860006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:23.462534002 +0000 UTC m=+1131.854341636" watchObservedRunningTime="2026-10-30 01:48:23.463860006 +0000 UTC m=+1131.855667639" Oct 30 01:48:23.724949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.724871 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.727259 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.727237 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\") pod \"maas-api-e2e-migrate-7664a237-64d6545d45-px4xp\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:23.932821 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:23.932788 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:24.241653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:24.241575 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp"] Oct 30 01:48:24.244816 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:24.244790 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf506cd24_b676_4179_84e3_4c8027bced34.slice/crio-22c2e4c1e8229eeb4dde23e242162818a9d66fdd2bdcb5e25f8ee03069bbbf60 WatchSource:0}: Error finding container 22c2e4c1e8229eeb4dde23e242162818a9d66fdd2bdcb5e25f8ee03069bbbf60: Status 404 returned error can't find the container with id 22c2e4c1e8229eeb4dde23e242162818a9d66fdd2bdcb5e25f8ee03069bbbf60 Oct 30 01:48:24.431166 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:24.431110 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" event={"ID":"f506cd24-b676-4179-84e3-4c8027bced34","Type":"ContainerStarted","Data":"22c2e4c1e8229eeb4dde23e242162818a9d66fdd2bdcb5e25f8ee03069bbbf60"} Oct 30 01:48:25.049720 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.049680 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z"] Oct 30 01:48:25.058210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.056831 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.073580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.069028 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z"] Oct 30 01:48:25.077352 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.077296 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-14b53b75\"" Oct 30 01:48:25.142499 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.142466 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume\") pod \"payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.142669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.142522 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fkb\" (UniqueName: \"kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb\") pod \"payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.164600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.164569 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-14b53b75-dockercfg-xfkdt\"" Oct 30 01:48:25.243924 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.243889 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume\") pod \"payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.244096 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.243938 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fkb\" (UniqueName: \"kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb\") pod \"payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.244627 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.244605 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume\") pod \"payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.266202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.266174 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fkb\" (UniqueName: \"kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb\") pod \"payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.411222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.410310 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:25.488889 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.488860 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" event={"ID":"f506cd24-b676-4179-84e3-4c8027bced34","Type":"ContainerStarted","Data":"9be2f74f73be0a1b7fa27f8f238425e7fcfae7ab3f8f1c49b70f49f0a095e97e"} Oct 30 01:48:25.489043 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.488999 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:25.541327 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.540343 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" podStartSLOduration=3.228050336 podStartE2EDuration="3.540314938s" totalImagesPullingTime="312.264602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:25.538344838 +0000 UTC m=+1133.930152488" watchObservedRunningTime="2026-10-30 01:48:25.540314938 +0000 UTC m=+1133.932122617" Oct 30 01:48:25.800496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:25.800403 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z"] Oct 30 01:48:25.803251 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:25.803215 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184439ed_5063_4b70_9961_b15f80245427.slice/crio-c1928cee13f948a85f509ddf9a5e64653e28b80b430711b3bc7beb628f6c5979 WatchSource:0}: Error finding container c1928cee13f948a85f509ddf9a5e64653e28b80b430711b3bc7beb628f6c5979: Status 404 returned error can't find the container with id c1928cee13f948a85f509ddf9a5e64653e28b80b430711b3bc7beb628f6c5979 Oct 30 01:48:26.462638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.462598 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz"] Oct 30 01:48:26.466368 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.466349 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.479935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.479912 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-61b6469a-serving-cert\"" Oct 30 01:48:26.484488 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.484459 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.484594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.484524 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.484679 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.484644 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf48h\" (UniqueName: \"kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.484944 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.484792 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.490448 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.489743 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz"] Oct 30 01:48:26.498316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.498288 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" event={"ID":"184439ed-5063-4b70-9961-b15f80245427","Type":"ContainerStarted","Data":"51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8"} Oct 30 01:48:26.498419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.498343 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" event={"ID":"184439ed-5063-4b70-9961-b15f80245427","Type":"ContainerStarted","Data":"c1928cee13f948a85f509ddf9a5e64653e28b80b430711b3bc7beb628f6c5979"} Oct 30 01:48:26.498610 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.498597 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:26.541836 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.541787 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" podStartSLOduration=1.54177482 podStartE2EDuration="1.54177482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:26.540769426 +0000 UTC m=+1134.932577058" watchObservedRunningTime="2026-10-30 01:48:26.54177482 +0000 UTC m=+1134.933582453" Oct 30 01:48:26.585773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.585740 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf48h\" (UniqueName: \"kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.585918 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.585818 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.585987 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.585972 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.586295 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.586032 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.586295 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:26.586188 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-serving-cert: secret "maas-api-e2e-ait-61b6469a-serving-cert" not found Oct 30 01:48:26.586295 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:26.586265 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls podName:79b09f9a-0507-41eb-bec1-57ac311ba0e7 nodeName:}" failed. No retries permitted until 2026-10-30 01:48:27.08624525 +0000 UTC m=+1135.478052873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls") pod "maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" (UID: "79b09f9a-0507-41eb-bec1-57ac311ba0e7") : secret "maas-api-e2e-ait-61b6469a-serving-cert" not found Oct 30 01:48:26.586721 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.586701 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.588570 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.588544 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:26.607561 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:26.607538 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf48h\" (UniqueName: \"kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:27.091844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:27.091792 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:27.094286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:27.094262 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\") pod \"maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:27.378912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:27.378801 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:27.689210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:27.689107 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz"] Oct 30 01:48:27.692708 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:27.692674 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b09f9a_0507_41eb_bec1_57ac311ba0e7.slice/crio-6c94fe8feec300c43497a7274d1eed7389e3c0a8c97ed7c902a54f9b7ec470b9 WatchSource:0}: Error finding container 6c94fe8feec300c43497a7274d1eed7389e3c0a8c97ed7c902a54f9b7ec470b9: Status 404 returned error can't find the container with id 6c94fe8feec300c43497a7274d1eed7389e3c0a8c97ed7c902a54f9b7ec470b9 Oct 30 01:48:28.164786 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.164756 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj"] Oct 30 01:48:28.168542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.168517 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.191933 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.191905 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-14b53b75-serving-cert\"" Oct 30 01:48:28.194338 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.194310 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj"] Oct 30 01:48:28.204844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.204821 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.204990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.204971 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.205055 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.205041 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8vg\" (UniqueName: \"kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.205096 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.205066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.306279 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.306195 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.306279 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.306247 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8vg\" (UniqueName: \"kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.306279 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.306267 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.306519 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.306311 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.306998 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.306971 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.308693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.308673 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.308805 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.308751 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.328112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.328082 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8vg\" (UniqueName: \"kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg\") pod \"maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.430076 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.430048 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:28.482220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.482188 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:28.513702 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.513663 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" event={"ID":"79b09f9a-0507-41eb-bec1-57ac311ba0e7","Type":"ContainerStarted","Data":"11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210"} Oct 30 01:48:28.514198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.513851 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" event={"ID":"79b09f9a-0507-41eb-bec1-57ac311ba0e7","Type":"ContainerStarted","Data":"6c94fe8feec300c43497a7274d1eed7389e3c0a8c97ed7c902a54f9b7ec470b9"} Oct 30 01:48:28.514198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.513918 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:28.565572 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.565479 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" podStartSLOduration=2.210866154 podStartE2EDuration="2.565461069s" totalImagesPullingTime="354.594915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:28.563581745 +0000 UTC m=+1136.955389379" watchObservedRunningTime="2026-10-30 01:48:28.565461069 +0000 UTC m=+1136.957268703" Oct 30 01:48:28.801652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:28.801618 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj"] Oct 30 01:48:28.805201 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:28.805166 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68de9e3_6960_4919_835d_89e8883ae560.slice/crio-e414ee62650bfc9d5b667c2d0a9de3089e7d36cf497bde41f9f8ea97c4acdcac WatchSource:0}: Error finding container e414ee62650bfc9d5b667c2d0a9de3089e7d36cf497bde41f9f8ea97c4acdcac: Status 404 returned error can't find the container with id e414ee62650bfc9d5b667c2d0a9de3089e7d36cf497bde41f9f8ea97c4acdcac Oct 30 01:48:29.521326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:29.521284 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" event={"ID":"c68de9e3-6960-4919-835d-89e8883ae560","Type":"ContainerStarted","Data":"7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9"} Oct 30 01:48:29.521506 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:29.521367 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" event={"ID":"c68de9e3-6960-4919-835d-89e8883ae560","Type":"ContainerStarted","Data":"e414ee62650bfc9d5b667c2d0a9de3089e7d36cf497bde41f9f8ea97c4acdcac"} Oct 30 01:48:29.521559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:29.521518 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:29.563615 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:29.563546 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" podStartSLOduration=1.169757712 podStartE2EDuration="1.563526116s" totalImagesPullingTime="393.768404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:29.5619556 +0000 UTC m=+1137.953763230" watchObservedRunningTime="2026-10-30 01:48:29.563526116 +0000 UTC m=+1137.955333814" Oct 30 01:48:31.504298 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:31.504220 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:32.420446 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:32.420419 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:32.420612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:32.420494 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:34.435415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:34.435385 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:34.530760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:34.528164 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:35.532944 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:35.532909 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:37.506960 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:37.506928 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:48:40.641706 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:40.641670 6275 generic.go:356] "Generic (PLEG): container finished" podID="67d45657-27b5-4070-aadc-b93e94487605" containerID="cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c" exitCode=0 Oct 30 01:48:40.642212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:40.641728 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" event={"ID":"67d45657-27b5-4070-aadc-b93e94487605","Type":"ContainerDied","Data":"cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c"} Oct 30 01:48:40.642297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:40.642266 6275 scope.go:122] "RemoveContainer" containerID="cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c" Oct 30 01:48:40.643647 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:40.643616 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:40.643729 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:40.643683 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:40.643729 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:40.643718 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:41.673097 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:41.673064 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" event={"ID":"67d45657-27b5-4070-aadc-b93e94487605","Type":"ContainerStarted","Data":"83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626"} Oct 30 01:48:42.608610 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:42.608567 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:42.609705 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:42.609386 6275 patch_prober.go:29] interesting pod/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:42.609705 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:42.609446 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:48:43.604299 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.604268 6275 patch_prober.go:29] interesting pod/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:43.604679 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.604321 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:48:43.685457 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.685374 6275 generic.go:356] "Generic (PLEG): container finished" podID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerID="f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8" exitCode=0 Oct 30 01:48:43.685457 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.685445 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" event={"ID":"7404d52a-793e-4bd1-a262-8ee7e49e0c8e","Type":"ContainerDied","Data":"f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8"} Oct 30 01:48:43.685857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.685841 6275 scope.go:122] "RemoveContainer" containerID="f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8" Oct 30 01:48:43.687063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.687038 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:43.687146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.687109 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:43.687188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:43.687163 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:44.323538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:44.323499 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw"] Oct 30 01:48:44.393148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:44.382962 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s"] Oct 30 01:48:44.604489 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:44.603840 6275 patch_prober.go:29] interesting pod/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:44.604489 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:44.603926 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:48:44.695872 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:44.695831 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" event={"ID":"7404d52a-793e-4bd1-a262-8ee7e49e0c8e","Type":"ContainerStarted","Data":"3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2"} Oct 30 01:48:45.283838 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:45.283809 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:45.289270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:45.289234 6275 patch_prober.go:29] interesting pod/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:45.289517 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:45.289487 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 01:48:45.603611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:45.603522 6275 patch_prober.go:29] interesting pod/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:45.603611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:45.603576 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:48:46.281565 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:46.281527 6275 patch_prober.go:29] interesting pod/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:46.282319 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:46.282282 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 01:48:46.604222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:46.604131 6275 patch_prober.go:29] interesting pod/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:46.604222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:46.604197 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:48:46.697548 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:46.697490 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:48:47.281785 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:47.281752 6275 patch_prober.go:29] interesting pod/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:47.282224 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:47.281806 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 01:48:47.388301 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:47.387379 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:47.608806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:47.608735 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:47.711979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:47.711952 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:48:48.281803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:48.281765 6275 patch_prober.go:29] interesting pod/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:48.282245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:48.281836 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 01:48:49.282084 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.282048 6275 patch_prober.go:29] interesting pod/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:49.282512 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.282103 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 01:48:49.324431 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.324395 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" podUID="d2f69095-f427-4566-ae48-72cf0af181c8" containerName="payload-processing" containerID="cri-o://b1c885fe3654535fa2c81febf464c237fcf0f0ddbe9239994f1f5f5ea5e51c00" gracePeriod=25 Oct 30 01:48:49.383353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.383311 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" podUID="ac2c120d-59cc-4e41-9105-44e64f37893d" containerName="payload-pre-processing" containerID="cri-o://a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3" gracePeriod=25 Oct 30 01:48:49.399021 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.398997 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:49.565911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.565816 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr"] Oct 30 01:48:49.724848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.724566 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:49.725452 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.725388 6275 generic.go:356] "Generic (PLEG): container finished" podID="d2f69095-f427-4566-ae48-72cf0af181c8" containerID="b1c885fe3654535fa2c81febf464c237fcf0f0ddbe9239994f1f5f5ea5e51c00" exitCode=0 Oct 30 01:48:49.725534 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.725478 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" event={"ID":"d2f69095-f427-4566-ae48-72cf0af181c8","Type":"ContainerDied","Data":"b1c885fe3654535fa2c81febf464c237fcf0f0ddbe9239994f1f5f5ea5e51c00"} Oct 30 01:48:49.726922 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.726900 6275 generic.go:356] "Generic (PLEG): container finished" podID="ac2c120d-59cc-4e41-9105-44e64f37893d" containerID="a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3" exitCode=0 Oct 30 01:48:49.727031 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.726941 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" Oct 30 01:48:49.727031 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.726987 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" event={"ID":"ac2c120d-59cc-4e41-9105-44e64f37893d","Type":"ContainerDied","Data":"a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3"} Oct 30 01:48:49.727113 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.727045 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s" event={"ID":"ac2c120d-59cc-4e41-9105-44e64f37893d","Type":"ContainerDied","Data":"c14015c06623a8481c8f26b9fa1f9f3432f6c84847550a0f477ba0051a2cf582"} Oct 30 01:48:49.727113 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.727076 6275 scope.go:122] "RemoveContainer" containerID="a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3" Oct 30 01:48:49.733349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.733319 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:49.737331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.737297 6275 scope.go:122] "RemoveContainer" containerID="a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3" Oct 30 01:48:49.738301 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:49.738265 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3\": container with ID starting with a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3 not found: ID does not exist" containerID="a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3" Oct 30 01:48:49.738410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.738303 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3"} err="failed to get container status \"a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3\": rpc error: code = NotFound desc = could not find container \"a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3\": container with ID starting with a006ede41d0ec9157a0ea61732b3f928c542cfcc5c511e327df15263e387e8e3 not found: ID does not exist" Oct 30 01:48:49.884344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884189 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs\" (UniqueName: \"kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs\") pod \"d2f69095-f427-4566-ae48-72cf0af181c8\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " Oct 30 01:48:49.884344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884295 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume\") pod \"ac2c120d-59cc-4e41-9105-44e64f37893d\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " Oct 30 01:48:49.884344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884327 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume\") pod \"d2f69095-f427-4566-ae48-72cf0af181c8\" (UID: \"d2f69095-f427-4566-ae48-72cf0af181c8\") " Oct 30 01:48:49.885142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884405 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q\" (UniqueName: \"kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q\") pod \"ac2c120d-59cc-4e41-9105-44e64f37893d\" (UID: \"ac2c120d-59cc-4e41-9105-44e64f37893d\") " Oct 30 01:48:49.885142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884660 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume" pod "ac2c120d-59cc-4e41-9105-44e64f37893d" (UID: "ac2c120d-59cc-4e41-9105-44e64f37893d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:49.885142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884769 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2c120d-59cc-4e41-9105-44e64f37893d-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:49.885142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.884902 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume" pod "d2f69095-f427-4566-ae48-72cf0af181c8" (UID: "d2f69095-f427-4566-ae48-72cf0af181c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:49.886684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.886657 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs" pod "d2f69095-f427-4566-ae48-72cf0af181c8" (UID: "d2f69095-f427-4566-ae48-72cf0af181c8"). InnerVolumeSpecName "kube-api-access-cgpbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:49.887202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.887182 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q" pod "ac2c120d-59cc-4e41-9105-44e64f37893d" (UID: "ac2c120d-59cc-4e41-9105-44e64f37893d"). InnerVolumeSpecName "kube-api-access-5hs7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:49.985732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.985695 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgpbs\" (UniqueName: \"kubernetes.io/projected/d2f69095-f427-4566-ae48-72cf0af181c8-kube-api-access-cgpbs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:49.985732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.985725 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f69095-f427-4566-ae48-72cf0af181c8-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:49.985732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:49.985736 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hs7q\" (UniqueName: \"kubernetes.io/projected/ac2c120d-59cc-4e41-9105-44e64f37893d-kube-api-access-5hs7q\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:50.069169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.069111 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s"] Oct 30 01:48:50.072558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.072531 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc6b04f0-7944b685f-45f4s"] Oct 30 01:48:50.285439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.285403 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:50.286458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.286439 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:48:50.742590 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.742459 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" Oct 30 01:48:50.742590 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.742468 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw" event={"ID":"d2f69095-f427-4566-ae48-72cf0af181c8","Type":"ContainerDied","Data":"69eeef149024e605686e976f8cac39baedab9b30594dc4348d30adfe2655a0d3"} Oct 30 01:48:50.742590 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.742540 6275 scope.go:122] "RemoveContainer" containerID="b1c885fe3654535fa2c81febf464c237fcf0f0ddbe9239994f1f5f5ea5e51c00" Oct 30 01:48:50.788417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.788383 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw"] Oct 30 01:48:50.791643 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:50.791614 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc6b04f0-857d8b75bd-bcvpw"] Oct 30 01:48:51.737968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:51.737935 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2c120d-59cc-4e41-9105-44e64f37893d" path="/var/lib/kubelet/pods/ac2c120d-59cc-4e41-9105-44e64f37893d/volumes" Oct 30 01:48:51.738386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:51.738367 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f69095-f427-4566-ae48-72cf0af181c8" path="/var/lib/kubelet/pods/d2f69095-f427-4566-ae48-72cf0af181c8/volumes" Oct 30 01:48:52.190686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.190654 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q"] Oct 30 01:48:52.191045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.190924 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" podUID="9b4f5c09-fbf8-4815-b8a2-62b0a000f512" containerName="maas-api" containerID="cri-o://c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4" gracePeriod=30 Oct 30 01:48:52.450375 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.447149 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:52.560371 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.560341 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp"] Oct 30 01:48:52.560604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.560580 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" podUID="f506cd24-b676-4179-84e3-4c8027bced34" containerName="maas-api" containerID="cri-o://9be2f74f73be0a1b7fa27f8f238425e7fcfae7ab3f8f1c49b70f49f0a095e97e" gracePeriod=30 Oct 30 01:48:52.614852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.614810 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8\" (UniqueName: \"kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8\") pod \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " Oct 30 01:48:52.615034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.614860 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs\") pod \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " Oct 30 01:48:52.615034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.614966 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls\") pod \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " Oct 30 01:48:52.615191 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.615048 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca\") pod \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\" (UID: \"9b4f5c09-fbf8-4815-b8a2-62b0a000f512\") " Oct 30 01:48:52.615493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.615462 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca" pod "9b4f5c09-fbf8-4815-b8a2-62b0a000f512" (UID: "9b4f5c09-fbf8-4815-b8a2-62b0a000f512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:52.617016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.616984 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls" pod "9b4f5c09-fbf8-4815-b8a2-62b0a000f512" (UID: "9b4f5c09-fbf8-4815-b8a2-62b0a000f512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:52.617016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.617007 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs" pod "9b4f5c09-fbf8-4815-b8a2-62b0a000f512" (UID: "9b4f5c09-fbf8-4815-b8a2-62b0a000f512"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:52.617200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.617025 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8" pod "9b4f5c09-fbf8-4815-b8a2-62b0a000f512" (UID: "9b4f5c09-fbf8-4815-b8a2-62b0a000f512"). InnerVolumeSpecName "kube-api-access-8dpf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:52.716596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.716462 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpf8\" (UniqueName: \"kubernetes.io/projected/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-kube-api-access-8dpf8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:52.716596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.716498 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:52.716596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.716516 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:52.716596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.716528 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4f5c09-fbf8-4815-b8a2-62b0a000f512-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:52.759661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.759622 6275 generic.go:356] "Generic (PLEG): container finished" podID="f506cd24-b676-4179-84e3-4c8027bced34" containerID="9be2f74f73be0a1b7fa27f8f238425e7fcfae7ab3f8f1c49b70f49f0a095e97e" exitCode=0 Oct 30 01:48:52.760050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.759693 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" event={"ID":"f506cd24-b676-4179-84e3-4c8027bced34","Type":"ContainerDied","Data":"9be2f74f73be0a1b7fa27f8f238425e7fcfae7ab3f8f1c49b70f49f0a095e97e"} Oct 30 01:48:52.761148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.761104 6275 generic.go:356] "Generic (PLEG): container finished" podID="9b4f5c09-fbf8-4815-b8a2-62b0a000f512" containerID="c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4" exitCode=0 Oct 30 01:48:52.761261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.761165 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" Oct 30 01:48:52.761261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.761233 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" event={"ID":"9b4f5c09-fbf8-4815-b8a2-62b0a000f512","Type":"ContainerDied","Data":"c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4"} Oct 30 01:48:52.761387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.761265 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q" event={"ID":"9b4f5c09-fbf8-4815-b8a2-62b0a000f512","Type":"ContainerDied","Data":"a5eb6c00f93076e1bfd85b441050ee57cc2ee07a44c955aec8a86485120156e8"} Oct 30 01:48:52.761387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.761287 6275 scope.go:122] "RemoveContainer" containerID="c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4" Oct 30 01:48:52.778686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.778665 6275 scope.go:122] "RemoveContainer" containerID="c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4" Oct 30 01:48:52.779013 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:52.778990 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4\": container with ID starting with c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4 not found: ID does not exist" containerID="c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4" Oct 30 01:48:52.779177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.779019 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4"} err="failed to get container status \"c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4\": rpc error: code = NotFound desc = could not find container \"c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4\": container with ID starting with c4f19ff67c76d18a9cb54b67120ce596c0d42439354d03a5c993410c0c6a71e4 not found: ID does not exist" Oct 30 01:48:52.812937 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.812911 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q"] Oct 30 01:48:52.814810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.814780 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc6b04f0-9558bdfbc-kkt2q"] Oct 30 01:48:52.817838 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.817818 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:52.918497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.918458 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs\") pod \"f506cd24-b676-4179-84e3-4c8027bced34\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " Oct 30 01:48:52.918698 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.918609 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca\") pod \"f506cd24-b676-4179-84e3-4c8027bced34\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " Oct 30 01:48:52.918698 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.918664 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\") pod \"f506cd24-b676-4179-84e3-4c8027bced34\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " Oct 30 01:48:52.918821 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.918705 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg\") pod \"f506cd24-b676-4179-84e3-4c8027bced34\" (UID: \"f506cd24-b676-4179-84e3-4c8027bced34\") " Oct 30 01:48:52.919031 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.918998 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca" pod "f506cd24-b676-4179-84e3-4c8027bced34" (UID: "f506cd24-b676-4179-84e3-4c8027bced34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:52.921308 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.921282 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs" pod "f506cd24-b676-4179-84e3-4c8027bced34" (UID: "f506cd24-b676-4179-84e3-4c8027bced34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:52.921457 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.921433 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls" pod "f506cd24-b676-4179-84e3-4c8027bced34" (UID: "f506cd24-b676-4179-84e3-4c8027bced34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:52.921552 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:52.921476 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg" pod "f506cd24-b676-4179-84e3-4c8027bced34" (UID: "f506cd24-b676-4179-84e3-4c8027bced34"). InnerVolumeSpecName "kube-api-access-59dtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:53.024877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.024753 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:53.024877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.024792 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f506cd24-b676-4179-84e3-4c8027bced34-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:53.024877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.024808 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f506cd24-b676-4179-84e3-4c8027bced34-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:53.024877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.024824 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59dtg\" (UniqueName: \"kubernetes.io/projected/f506cd24-b676-4179-84e3-4c8027bced34-kube-api-access-59dtg\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:53.737877 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.737843 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4f5c09-fbf8-4815-b8a2-62b0a000f512" path="/var/lib/kubelet/pods/9b4f5c09-fbf8-4815-b8a2-62b0a000f512/volumes" Oct 30 01:48:53.766887 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.766859 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" Oct 30 01:48:53.767331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.766894 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp" event={"ID":"f506cd24-b676-4179-84e3-4c8027bced34","Type":"ContainerDied","Data":"22c2e4c1e8229eeb4dde23e242162818a9d66fdd2bdcb5e25f8ee03069bbbf60"} Oct 30 01:48:53.767331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.766956 6275 scope.go:122] "RemoveContainer" containerID="9be2f74f73be0a1b7fa27f8f238425e7fcfae7ab3f8f1c49b70f49f0a095e97e" Oct 30 01:48:53.818207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.818176 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp"] Oct 30 01:48:53.828573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:53.828546 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7664a237-64d6545d45-px4xp"] Oct 30 01:48:54.238910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.238872 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2"] Oct 30 01:48:54.239555 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239535 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ac2c120d-59cc-4e41-9105-44e64f37893d" containerName="payload-pre-processing" Oct 30 01:48:54.239658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239556 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2c120d-59cc-4e41-9105-44e64f37893d" containerName="payload-pre-processing" Oct 30 01:48:54.239658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239570 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f506cd24-b676-4179-84e3-4c8027bced34" containerName="maas-api" Oct 30 01:48:54.239658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239578 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f506cd24-b676-4179-84e3-4c8027bced34" containerName="maas-api" Oct 30 01:48:54.239658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239601 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d2f69095-f427-4566-ae48-72cf0af181c8" containerName="payload-processing" Oct 30 01:48:54.239658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239609 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f69095-f427-4566-ae48-72cf0af181c8" containerName="payload-processing" Oct 30 01:48:54.239810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239669 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9b4f5c09-fbf8-4815-b8a2-62b0a000f512" containerName="maas-api" Oct 30 01:48:54.239810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239677 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4f5c09-fbf8-4815-b8a2-62b0a000f512" containerName="maas-api" Oct 30 01:48:54.239810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239799 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4f5c09-fbf8-4815-b8a2-62b0a000f512" containerName="maas-api" Oct 30 01:48:54.239902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239812 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2c120d-59cc-4e41-9105-44e64f37893d" containerName="payload-pre-processing" Oct 30 01:48:54.239902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239823 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f506cd24-b676-4179-84e3-4c8027bced34" containerName="maas-api" Oct 30 01:48:54.239902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.239832 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f69095-f427-4566-ae48-72cf0af181c8" containerName="payload-processing" Oct 30 01:48:54.243446 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.243424 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.258983 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.258963 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-719029-openshift-default-dockercfg-9jg4v\"" Oct 30 01:48:54.275047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.275014 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2"] Oct 30 01:48:54.341500 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.341453 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.341834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.341812 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.341982 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.341967 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.342090 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.342077 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.342248 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.342230 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrzb\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.342365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.342352 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.342476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.342463 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.342627 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.342613 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.342846 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.342821 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444450 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444508 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444542 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444564 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrzb\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444725 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444810 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444904 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.444968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444927 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.445072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.444985 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.445072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.445053 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.445462 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.445435 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.445790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.445767 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.445880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.445846 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.446063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.446042 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.447132 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.447095 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.447222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.447159 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.465931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.465900 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrzb\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.466045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.465969 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.556423 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.556391 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:54.566194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.566162 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" podUID="ea416368-c50c-4926-aafb-b6ae7ec0c790" containerName="payload-processing" containerID="cri-o://666bc3e9a558e95040ca2a4a91e5c60386dfaf82bed21dfbd9111c6197457773" gracePeriod=25 Oct 30 01:48:54.737167 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.735727 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz"] Oct 30 01:48:54.737167 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.735978 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" podUID="79b09f9a-0507-41eb-bec1-57ac311ba0e7" containerName="maas-api" containerID="cri-o://11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210" gracePeriod=30 Oct 30 01:48:54.777879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.777734 6275 generic.go:356] "Generic (PLEG): container finished" podID="ea416368-c50c-4926-aafb-b6ae7ec0c790" containerID="666bc3e9a558e95040ca2a4a91e5c60386dfaf82bed21dfbd9111c6197457773" exitCode=0 Oct 30 01:48:54.777879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.777808 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" event={"ID":"ea416368-c50c-4926-aafb-b6ae7ec0c790","Type":"ContainerDied","Data":"666bc3e9a558e95040ca2a4a91e5c60386dfaf82bed21dfbd9111c6197457773"} Oct 30 01:48:54.858505 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.858472 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:54.909384 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.909348 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2"] Oct 30 01:48:54.914861 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:54.914836 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b15538e_985f_47e5_9ebf_f4762969d4b3.slice/crio-8129a3c911e61c25b967d2acc2acc20399bf15688ea8631dc82c56dff91549be WatchSource:0}: Error finding container 8129a3c911e61c25b967d2acc2acc20399bf15688ea8631dc82c56dff91549be: Status 404 returned error can't find the container with id 8129a3c911e61c25b967d2acc2acc20399bf15688ea8631dc82c56dff91549be Oct 30 01:48:54.917482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.917450 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:54.917558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.917526 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:54.917591 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.917556 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:54.994016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:54.993992 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:55.054911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.054874 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f\" (UniqueName: \"kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f\") pod \"ea416368-c50c-4926-aafb-b6ae7ec0c790\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " Oct 30 01:48:55.055083 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.054985 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume\") pod \"ea416368-c50c-4926-aafb-b6ae7ec0c790\" (UID: \"ea416368-c50c-4926-aafb-b6ae7ec0c790\") " Oct 30 01:48:55.056380 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.056334 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume" pod "ea416368-c50c-4926-aafb-b6ae7ec0c790" (UID: "ea416368-c50c-4926-aafb-b6ae7ec0c790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:55.058732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.058708 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f" pod "ea416368-c50c-4926-aafb-b6ae7ec0c790" (UID: "ea416368-c50c-4926-aafb-b6ae7ec0c790"). InnerVolumeSpecName "kube-api-access-7ns8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:55.156199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.156083 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\") pod \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " Oct 30 01:48:55.156366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.156234 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca\") pod \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " Oct 30 01:48:55.156366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.156281 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs\") pod \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " Oct 30 01:48:55.156366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.156345 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h\" (UniqueName: \"kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h\") pod \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\" (UID: \"79b09f9a-0507-41eb-bec1-57ac311ba0e7\") " Oct 30 01:48:55.156954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.156653 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ns8f\" (UniqueName: \"kubernetes.io/projected/ea416368-c50c-4926-aafb-b6ae7ec0c790-kube-api-access-7ns8f\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.156954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.156678 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea416368-c50c-4926-aafb-b6ae7ec0c790-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.157169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.157144 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca" pod "79b09f9a-0507-41eb-bec1-57ac311ba0e7" (UID: "79b09f9a-0507-41eb-bec1-57ac311ba0e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:55.160276 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.160241 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs" pod "79b09f9a-0507-41eb-bec1-57ac311ba0e7" (UID: "79b09f9a-0507-41eb-bec1-57ac311ba0e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:55.160800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.160764 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls" pod "79b09f9a-0507-41eb-bec1-57ac311ba0e7" (UID: "79b09f9a-0507-41eb-bec1-57ac311ba0e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:55.163315 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.163289 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h" pod "79b09f9a-0507-41eb-bec1-57ac311ba0e7" (UID: "79b09f9a-0507-41eb-bec1-57ac311ba0e7"). InnerVolumeSpecName "kube-api-access-bf48h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:55.257703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.257666 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.257703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.257701 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b09f9a-0507-41eb-bec1-57ac311ba0e7-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.257950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.257716 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b09f9a-0507-41eb-bec1-57ac311ba0e7-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.257950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.257729 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf48h\" (UniqueName: \"kubernetes.io/projected/79b09f9a-0507-41eb-bec1-57ac311ba0e7-kube-api-access-bf48h\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.737254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.737179 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f506cd24-b676-4179-84e3-4c8027bced34" path="/var/lib/kubelet/pods/f506cd24-b676-4179-84e3-4c8027bced34/volumes" Oct 30 01:48:55.786242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.786209 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" Oct 30 01:48:55.786666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.786216 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr" event={"ID":"ea416368-c50c-4926-aafb-b6ae7ec0c790","Type":"ContainerDied","Data":"b17022635ad2eb31d05a97dd2afaabdea427fc53abba732db7bb9fbfd91de711"} Oct 30 01:48:55.786666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.786364 6275 scope.go:122] "RemoveContainer" containerID="666bc3e9a558e95040ca2a4a91e5c60386dfaf82bed21dfbd9111c6197457773" Oct 30 01:48:55.787804 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.787777 6275 generic.go:356] "Generic (PLEG): container finished" podID="79b09f9a-0507-41eb-bec1-57ac311ba0e7" containerID="11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210" exitCode=0 Oct 30 01:48:55.787927 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.787828 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" Oct 30 01:48:55.787927 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.787859 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" event={"ID":"79b09f9a-0507-41eb-bec1-57ac311ba0e7","Type":"ContainerDied","Data":"11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210"} Oct 30 01:48:55.787927 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.787915 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz" event={"ID":"79b09f9a-0507-41eb-bec1-57ac311ba0e7","Type":"ContainerDied","Data":"6c94fe8feec300c43497a7274d1eed7389e3c0a8c97ed7c902a54f9b7ec470b9"} Oct 30 01:48:55.789629 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.789598 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" event={"ID":"6b15538e-985f-47e5-9ebf-f4762969d4b3","Type":"ContainerStarted","Data":"cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330"} Oct 30 01:48:55.789755 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.789636 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" event={"ID":"6b15538e-985f-47e5-9ebf-f4762969d4b3","Type":"ContainerStarted","Data":"8129a3c911e61c25b967d2acc2acc20399bf15688ea8631dc82c56dff91549be"} Oct 30 01:48:55.795746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.795727 6275 scope.go:122] "RemoveContainer" containerID="11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210" Oct 30 01:48:55.804326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.804309 6275 scope.go:122] "RemoveContainer" containerID="11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210" Oct 30 01:48:55.804576 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:55.804559 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210\": container with ID starting with 11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210 not found: ID does not exist" containerID="11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210" Oct 30 01:48:55.804624 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.804585 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210"} err="failed to get container status \"11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210\": rpc error: code = NotFound desc = could not find container \"11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210\": container with ID starting with 11d06e74532c7c3691bae6a3f6138f9600b36bc88039e2a50313dc1308c76210 not found: ID does not exist" Oct 30 01:48:55.832361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.832338 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr"] Oct 30 01:48:55.842919 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.842896 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7664a237-577f9d5b4-kzhqr"] Oct 30 01:48:55.879271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.879205 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz"] Oct 30 01:48:55.881532 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.881505 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61b6469a-75cd7b4855-wn4kz"] Oct 30 01:48:55.918670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:55.918619 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podStartSLOduration=1.918601724 podStartE2EDuration="1.918601724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:55.916799531 +0000 UTC m=+1164.308607165" watchObservedRunningTime="2026-10-30 01:48:55.918601724 +0000 UTC m=+1164.310409362" Oct 30 01:48:56.557108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:56.557077 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:56.562106 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:56.562077 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:56.798392 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:56.798357 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:56.799512 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:56.799494 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:48:56.896294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:56.896211 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z"] Oct 30 01:48:57.082153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.074987 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj"] Oct 30 01:48:57.082153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.076056 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" podUID="c68de9e3-6960-4919-835d-89e8883ae560" containerName="maas-api" containerID="cri-o://7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9" gracePeriod=30 Oct 30 01:48:57.320088 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.320061 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:57.393497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.393466 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls\") pod \"c68de9e3-6960-4919-835d-89e8883ae560\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " Oct 30 01:48:57.393676 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.393504 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg\" (UniqueName: \"kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg\") pod \"c68de9e3-6960-4919-835d-89e8883ae560\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " Oct 30 01:48:57.393676 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.393590 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs\") pod \"c68de9e3-6960-4919-835d-89e8883ae560\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " Oct 30 01:48:57.393676 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.393632 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca\") pod \"c68de9e3-6960-4919-835d-89e8883ae560\" (UID: \"c68de9e3-6960-4919-835d-89e8883ae560\") " Oct 30 01:48:57.394162 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.394115 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca" pod "c68de9e3-6960-4919-835d-89e8883ae560" (UID: "c68de9e3-6960-4919-835d-89e8883ae560"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:57.395663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.395633 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg" pod "c68de9e3-6960-4919-835d-89e8883ae560" (UID: "c68de9e3-6960-4919-835d-89e8883ae560"). InnerVolumeSpecName "kube-api-access-pm8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:57.395776 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.395730 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls" pod "c68de9e3-6960-4919-835d-89e8883ae560" (UID: "c68de9e3-6960-4919-835d-89e8883ae560"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:57.396050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.396035 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs" pod "c68de9e3-6960-4919-835d-89e8883ae560" (UID: "c68de9e3-6960-4919-835d-89e8883ae560"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:57.494880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.494781 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:57.494880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.494825 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm8vg\" (UniqueName: \"kubernetes.io/projected/c68de9e3-6960-4919-835d-89e8883ae560-kube-api-access-pm8vg\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:57.494880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.494843 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68de9e3-6960-4919-835d-89e8883ae560-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:57.494880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.494860 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68de9e3-6960-4919-835d-89e8883ae560-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:48:57.737856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.737824 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b09f9a-0507-41eb-bec1-57ac311ba0e7" path="/var/lib/kubelet/pods/79b09f9a-0507-41eb-bec1-57ac311ba0e7/volumes" Oct 30 01:48:57.738241 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.738226 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea416368-c50c-4926-aafb-b6ae7ec0c790" path="/var/lib/kubelet/pods/ea416368-c50c-4926-aafb-b6ae7ec0c790/volumes" Oct 30 01:48:57.804081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.804046 6275 generic.go:356] "Generic (PLEG): container finished" podID="c68de9e3-6960-4919-835d-89e8883ae560" containerID="7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9" exitCode=0 Oct 30 01:48:57.804493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.804094 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" Oct 30 01:48:57.804493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.804154 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" event={"ID":"c68de9e3-6960-4919-835d-89e8883ae560","Type":"ContainerDied","Data":"7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9"} Oct 30 01:48:57.804493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.804204 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj" event={"ID":"c68de9e3-6960-4919-835d-89e8883ae560","Type":"ContainerDied","Data":"e414ee62650bfc9d5b667c2d0a9de3089e7d36cf497bde41f9f8ea97c4acdcac"} Oct 30 01:48:57.804493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.804230 6275 scope.go:122] "RemoveContainer" containerID="7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9" Oct 30 01:48:57.816191 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.816173 6275 scope.go:122] "RemoveContainer" containerID="7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9" Oct 30 01:48:57.816434 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:48:57.816415 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9\": container with ID starting with 7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9 not found: ID does not exist" containerID="7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9" Oct 30 01:48:57.816495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.816447 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9"} err="failed to get container status \"7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9\": rpc error: code = NotFound desc = could not find container \"7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9\": container with ID starting with 7068115e9bfe683aa6d0368f26512051edf2b29baf7f17dd071ec6d0a8fc8bc9 not found: ID does not exist" Oct 30 01:48:57.851359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.851322 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj"] Oct 30 01:48:57.858915 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.858886 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14b53b75-7c7b7b86d7-nbxcj"] Oct 30 01:48:57.886484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.886454 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r"] Oct 30 01:48:57.886696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:57.886669 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" containerID="cri-o://83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626" gracePeriod=30 Oct 30 01:48:59.243176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243142 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld"] Oct 30 01:48:59.243773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243751 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c68de9e3-6960-4919-835d-89e8883ae560" containerName="maas-api" Oct 30 01:48:59.243773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243774 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68de9e3-6960-4919-835d-89e8883ae560" containerName="maas-api" Oct 30 01:48:59.243932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243819 6275 cpu_manager.go:414] "containerMap: removing container" podUID="79b09f9a-0507-41eb-bec1-57ac311ba0e7" containerName="maas-api" Oct 30 01:48:59.243932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243830 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b09f9a-0507-41eb-bec1-57ac311ba0e7" containerName="maas-api" Oct 30 01:48:59.243932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243854 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ea416368-c50c-4926-aafb-b6ae7ec0c790" containerName="payload-processing" Oct 30 01:48:59.243932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243862 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea416368-c50c-4926-aafb-b6ae7ec0c790" containerName="payload-processing" Oct 30 01:48:59.244211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243977 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b09f9a-0507-41eb-bec1-57ac311ba0e7" containerName="maas-api" Oct 30 01:48:59.244211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.243994 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea416368-c50c-4926-aafb-b6ae7ec0c790" containerName="payload-processing" Oct 30 01:48:59.244211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.244006 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68de9e3-6960-4919-835d-89e8883ae560" containerName="maas-api" Oct 30 01:48:59.246887 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.246866 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.271567 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.271547 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b93d51-openshift-default-dockercfg-qh2lt\"" Oct 30 01:48:59.296426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.296392 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld"] Oct 30 01:48:59.314879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.314848 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.314912 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315056 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315010 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315100 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315099 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315147 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315197 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315223 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqq7\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.315261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.315249 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.416936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.416894 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417115 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.416973 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417115 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417009 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417115 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417054 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417115 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417089 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417367 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417141 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417367 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417194 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417367 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417221 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqq7\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417367 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417247 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417367 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417312 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417630 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417406 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417679 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417833 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417812 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.417903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.417887 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.420042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.420012 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.420261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.420239 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.459578 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.459548 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqq7\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.465955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.465923 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.558493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.558460 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:48:59.737829 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.737784 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68de9e3-6960-4919-835d-89e8883ae560" path="/var/lib/kubelet/pods/c68de9e3-6960-4919-835d-89e8883ae560/volumes" Oct 30 01:48:59.899515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.899441 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld"] Oct 30 01:48:59.902679 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:48:59.902653 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57544d9_f1dc_492f_aa75_d00704637f9b.slice/crio-99946bec5d5f18461f7ce098b8ff087c16566d33cb7bca32ff6154db5ff70d10 WatchSource:0}: Error finding container 99946bec5d5f18461f7ce098b8ff087c16566d33cb7bca32ff6154db5ff70d10: Status 404 returned error can't find the container with id 99946bec5d5f18461f7ce098b8ff087c16566d33cb7bca32ff6154db5ff70d10 Oct 30 01:48:59.905019 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.904986 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:59.905095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.905051 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:48:59.905095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:48:59.905085 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:00.233872 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:00.233786 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh"] Oct 30 01:49:00.234115 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:00.234066 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" containerID="cri-o://3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2" gracePeriod=30 Oct 30 01:49:00.822808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:00.822769 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" event={"ID":"d57544d9-f1dc-492f-aa75-d00704637f9b","Type":"ContainerStarted","Data":"9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc"} Oct 30 01:49:00.823220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:00.822823 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" event={"ID":"d57544d9-f1dc-492f-aa75-d00704637f9b","Type":"ContainerStarted","Data":"99946bec5d5f18461f7ce098b8ff087c16566d33cb7bca32ff6154db5ff70d10"} Oct 30 01:49:00.868151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:00.868089 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podStartSLOduration=1.868074456 podStartE2EDuration="1.868074456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:00.867116502 +0000 UTC m=+1169.258924136" watchObservedRunningTime="2026-10-30 01:49:00.868074456 +0000 UTC m=+1169.259882088" Oct 30 01:49:01.420859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.420809 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7"] Oct 30 01:49:01.423988 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.423964 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.437938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.437909 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c40a89-openshift-default-dockercfg-vvdq4\"" Oct 30 01:49:01.445533 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445505 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttl8\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445673 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445545 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445673 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445650 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445691 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445770 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445793 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445875 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445906 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.445949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.445940 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.451790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.451757 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7"] Oct 30 01:49:01.547063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547022 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547117 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547179 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547217 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547239 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547273 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547352 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttl8\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547381 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547580 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547630 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547635 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547769 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547830 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.547961 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.547941 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.549993 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.549965 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.550214 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.550198 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.559190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.559169 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:01.564662 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.564642 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:01.568890 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.568868 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.569631 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.569165 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttl8\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.739477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.739395 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:01.828293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.828258 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:01.829332 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.829312 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:01.840054 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.840023 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8"] Oct 30 01:49:01.843613 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.843591 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:01.873849 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.873819 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8"] Oct 30 01:49:01.885719 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.885694 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-78d5ff-serving-cert\"" Oct 30 01:49:01.896828 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.896781 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" podUID="184439ed-5063-4b70-9961-b15f80245427" containerName="payload-processing" containerID="cri-o://51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8" gracePeriod=25 Oct 30 01:49:01.953735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.953702 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzn9\" (UniqueName: \"kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:01.953876 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.953740 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:01.953876 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.953772 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:01.953968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:01.953934 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.064210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.061782 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7"] Oct 30 01:49:02.066242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.066201 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzn9\" (UniqueName: \"kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.066424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.066256 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.066424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.066306 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.066424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.066375 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.067267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.067243 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.072390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.071578 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.072515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.072468 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.076172 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:02.074496 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b68df8f_6283_4f8e_9e70_22ac355cc809.slice/crio-a6eb757750fa3f3a2b67f388e3a5260f0f68203c5239ad1252fa5f1af030add5 WatchSource:0}: Error finding container a6eb757750fa3f3a2b67f388e3a5260f0f68203c5239ad1252fa5f1af030add5: Status 404 returned error can't find the container with id a6eb757750fa3f3a2b67f388e3a5260f0f68203c5239ad1252fa5f1af030add5 Oct 30 01:49:02.078331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.077548 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:02.078331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.077643 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:02.078331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.077685 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:02.093351 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.093321 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzn9\" (UniqueName: \"kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9\") pod \"maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.162429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.162327 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:02.204620 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.204377 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:49:02.270170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.269229 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume\") pod \"184439ed-5063-4b70-9961-b15f80245427\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " Oct 30 01:49:02.271345 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.271221 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume" pod "184439ed-5063-4b70-9961-b15f80245427" (UID: "184439ed-5063-4b70-9961-b15f80245427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:02.271553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.271533 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb\" (UniqueName: \"kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb\") pod \"184439ed-5063-4b70-9961-b15f80245427\" (UID: \"184439ed-5063-4b70-9961-b15f80245427\") " Oct 30 01:49:02.272175 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.272142 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184439ed-5063-4b70-9961-b15f80245427-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:02.274239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.274214 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb" pod "184439ed-5063-4b70-9961-b15f80245427" (UID: "184439ed-5063-4b70-9961-b15f80245427"). InnerVolumeSpecName "kube-api-access-f8fkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:02.373498 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.373464 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8fkb\" (UniqueName: \"kubernetes.io/projected/184439ed-5063-4b70-9961-b15f80245427-kube-api-access-f8fkb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:02.494477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.494443 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8"] Oct 30 01:49:02.498098 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:02.498071 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e176405_a61d_4763_8f40_1c40de877bb2.slice/crio-87ea6717e2cbd393427cf015c97ba870aa74bb16e35fbf830acd6f5e1b23c812 WatchSource:0}: Error finding container 87ea6717e2cbd393427cf015c97ba870aa74bb16e35fbf830acd6f5e1b23c812: Status 404 returned error can't find the container with id 87ea6717e2cbd393427cf015c97ba870aa74bb16e35fbf830acd6f5e1b23c812 Oct 30 01:49:02.713445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.713408 6275 patch_prober.go:29] interesting pod/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:49:02.713584 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.713466 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:49:02.834848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.834817 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" event={"ID":"6b68df8f-6283-4f8e-9e70-22ac355cc809","Type":"ContainerStarted","Data":"a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b"} Oct 30 01:49:02.835285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.834877 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" event={"ID":"6b68df8f-6283-4f8e-9e70-22ac355cc809","Type":"ContainerStarted","Data":"a6eb757750fa3f3a2b67f388e3a5260f0f68203c5239ad1252fa5f1af030add5"} Oct 30 01:49:02.836200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.836172 6275 generic.go:356] "Generic (PLEG): container finished" podID="184439ed-5063-4b70-9961-b15f80245427" containerID="51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8" exitCode=0 Oct 30 01:49:02.836321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.836224 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" Oct 30 01:49:02.836321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.836250 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" event={"ID":"184439ed-5063-4b70-9961-b15f80245427","Type":"ContainerDied","Data":"51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8"} Oct 30 01:49:02.836321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.836292 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z" event={"ID":"184439ed-5063-4b70-9961-b15f80245427","Type":"ContainerDied","Data":"c1928cee13f948a85f509ddf9a5e64653e28b80b430711b3bc7beb628f6c5979"} Oct 30 01:49:02.836321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.836315 6275 scope.go:122] "RemoveContainer" containerID="51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8" Oct 30 01:49:02.837946 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.837924 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" event={"ID":"9e176405-a61d-4763-8f40-1c40de877bb2","Type":"ContainerStarted","Data":"87ea6717e2cbd393427cf015c97ba870aa74bb16e35fbf830acd6f5e1b23c812"} Oct 30 01:49:02.850005 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.849985 6275 scope.go:122] "RemoveContainer" containerID="51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8" Oct 30 01:49:02.850278 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:02.850260 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8\": container with ID starting with 51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8 not found: ID does not exist" containerID="51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8" Oct 30 01:49:02.850323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.850288 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8"} err="failed to get container status \"51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8\": rpc error: code = NotFound desc = could not find container \"51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8\": container with ID starting with 51411104cfb30fdd6bb4ca6ad2b93c64632198582dcee6156e60ed08a46213c8 not found: ID does not exist" Oct 30 01:49:02.878692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.878616 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podStartSLOduration=1.878603766 podStartE2EDuration="1.878603766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:02.877348067 +0000 UTC m=+1171.269155700" watchObservedRunningTime="2026-10-30 01:49:02.878603766 +0000 UTC m=+1171.270411399" Oct 30 01:49:02.912696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.912670 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z"] Oct 30 01:49:02.916581 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:02.916560 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14b53b75-66cd655bd8-rqk8z"] Oct 30 01:49:03.159684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.159656 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:49:03.281158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281107 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281159 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281327 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281202 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281327 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281267 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281327 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281300 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281360 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281382 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:03.281453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281395 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281477 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281508 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs\") pod \"67d45657-27b5-4070-aadc-b93e94487605\" (UID: \"67d45657-27b5-4070-aadc-b93e94487605\") " Oct 30 01:49:03.281707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281676 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:03.281767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281744 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:03.281881 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.281862 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:03.282070 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.282052 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d45657-27b5-4070-aadc-b93e94487605-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.282199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.282072 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:03.282199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.282088 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.282199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.282103 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.282199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.282145 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.283939 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.283920 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:03.283939 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.283926 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:03.284030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.283984 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "kube-api-access-lzgts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:03.284065 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.284033 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo" pod "67d45657-27b5-4070-aadc-b93e94487605" (UID: "67d45657-27b5-4070-aadc-b93e94487605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:49:03.387781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.387704 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d45657-27b5-4070-aadc-b93e94487605-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.387781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.387732 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.387781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.387743 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgts\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-kube-api-access-lzgts\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.387781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.387753 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d45657-27b5-4070-aadc-b93e94487605-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.387781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.387761 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d45657-27b5-4070-aadc-b93e94487605-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:03.737784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.737697 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184439ed-5063-4b70-9961-b15f80245427" path="/var/lib/kubelet/pods/184439ed-5063-4b70-9961-b15f80245427/volumes" Oct 30 01:49:03.739667 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.739645 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:03.745921 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.745902 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:03.814806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.814722 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg"] Oct 30 01:49:03.815345 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815327 6275 cpu_manager.go:414] "containerMap: removing container" podUID="184439ed-5063-4b70-9961-b15f80245427" containerName="payload-processing" Oct 30 01:49:03.815400 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815348 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184439ed-5063-4b70-9961-b15f80245427" containerName="payload-processing" Oct 30 01:49:03.815400 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815387 6275 cpu_manager.go:414] "containerMap: removing container" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" Oct 30 01:49:03.815400 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815395 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" Oct 30 01:49:03.815516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815504 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" Oct 30 01:49:03.815550 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815521 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="184439ed-5063-4b70-9961-b15f80245427" containerName="payload-processing" Oct 30 01:49:03.815550 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.815535 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" Oct 30 01:49:03.818942 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.818921 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.843093 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.843067 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg"] Oct 30 01:49:03.845954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.845911 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" event={"ID":"9e176405-a61d-4763-8f40-1c40de877bb2","Type":"ContainerStarted","Data":"c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c"} Oct 30 01:49:03.846074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.846014 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:03.849407 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.849380 6275 generic.go:356] "Generic (PLEG): container finished" podID="67d45657-27b5-4070-aadc-b93e94487605" containerID="83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626" exitCode=0 Oct 30 01:49:03.849615 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.849599 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" Oct 30 01:49:03.850085 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.850065 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" event={"ID":"67d45657-27b5-4070-aadc-b93e94487605","Type":"ContainerDied","Data":"83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626"} Oct 30 01:49:03.850690 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.850110 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r" event={"ID":"67d45657-27b5-4070-aadc-b93e94487605","Type":"ContainerDied","Data":"b023124921b36acbdb8d47c2cc4987d4cdff07e6e61eb896c2043b31062cb73b"} Oct 30 01:49:03.850690 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.850195 6275 scope.go:122] "RemoveContainer" containerID="83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626" Oct 30 01:49:03.854520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.854497 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:03.855315 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.855295 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:03.863520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.863502 6275 scope.go:122] "RemoveContainer" containerID="cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c" Oct 30 01:49:03.872707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.872684 6275 scope.go:122] "RemoveContainer" containerID="83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626" Oct 30 01:49:03.873034 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:03.873009 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626\": container with ID starting with 83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626 not found: ID does not exist" containerID="83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626" Oct 30 01:49:03.873145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.873046 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626"} err="failed to get container status \"83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626\": rpc error: code = NotFound desc = could not find container \"83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626\": container with ID starting with 83f0ab6a32da7cc714777e2c566f9455b84e62df693bae4e484b8d95a28d9626 not found: ID does not exist" Oct 30 01:49:03.873145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.873072 6275 scope.go:122] "RemoveContainer" containerID="cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c" Oct 30 01:49:03.873414 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:03.873391 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c\": container with ID starting with cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c not found: ID does not exist" containerID="cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c" Oct 30 01:49:03.873481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.873420 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c"} err="failed to get container status \"cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c\": rpc error: code = NotFound desc = could not find container \"cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c\": container with ID starting with cc87d8cc5821425885910fb4ff9036d4bcaca5b549e3a18086c5943fa45e963c not found: ID does not exist" Oct 30 01:49:03.874712 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.874692 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-719029-serving-cert\"" Oct 30 01:49:03.894083 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.894051 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpdv\" (UniqueName: \"kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.894477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.894257 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.894477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.894400 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.894654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.894504 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.902210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.902174 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" podStartSLOduration=2.58154406 podStartE2EDuration="2.902162672s" totalImagesPullingTime="320.618612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:03.901660812 +0000 UTC m=+1172.293468446" watchObservedRunningTime="2026-10-30 01:49:03.902162672 +0000 UTC m=+1172.293970302" Oct 30 01:49:03.946445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.946412 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r"] Oct 30 01:49:03.953401 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.953371 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-61b6469a-openshift-default-7879947fc5-rqm7r"] Oct 30 01:49:03.996444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.996345 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpdv\" (UniqueName: \"kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.996444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.996426 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.996692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.996503 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.996750 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.996720 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.997242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.997218 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:03.999852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:03.999825 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:04.000095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.000073 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:04.027174 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.027143 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpdv\" (UniqueName: \"kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv\") pod \"maas-api-e2e-worker-w2-719029-64d78488-qbspg\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:04.143856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.143792 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:04.462065 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.462032 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg"] Oct 30 01:49:04.465537 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:04.465513 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0940d0_a9fd_44cf_8ce5_2b1f8f1eeeda.slice/crio-42e4f77bc26009621f3069cbab04440ae80bd3be5f9c69c10c64010da44001fa WatchSource:0}: Error finding container 42e4f77bc26009621f3069cbab04440ae80bd3be5f9c69c10c64010da44001fa: Status 404 returned error can't find the container with id 42e4f77bc26009621f3069cbab04440ae80bd3be5f9c69c10c64010da44001fa Oct 30 01:49:04.861814 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.861777 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" event={"ID":"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda","Type":"ContainerStarted","Data":"034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6"} Oct 30 01:49:04.862317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.861839 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" event={"ID":"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda","Type":"ContainerStarted","Data":"42e4f77bc26009621f3069cbab04440ae80bd3be5f9c69c10c64010da44001fa"} Oct 30 01:49:04.862317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.862165 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:04.906067 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:04.906015 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" podStartSLOduration=1.650750801 podStartE2EDuration="1.906001363s" totalImagesPullingTime="255.250562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:04.904097762 +0000 UTC m=+1173.295905396" watchObservedRunningTime="2026-10-30 01:49:04.906001363 +0000 UTC m=+1173.297808995" Oct 30 01:49:05.286998 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.286970 6275 patch_prober.go:29] interesting pod/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:49:05.287142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.287019 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 01:49:05.501742 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.501679 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:49:05.611441 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611401 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611590 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611554 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611612 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611637 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:05.611779 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611729 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611779 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611764 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611878 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611814 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611878 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611843 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.611878 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611874 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.612028 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611913 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert\") pod \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\" (UID: \"7404d52a-793e-4bd1-a262-8ee7e49e0c8e\") " Oct 30 01:49:05.612028 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.611913 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:05.612158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612061 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:05.612266 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612233 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:05.612384 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612366 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:05.612439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612415 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.612439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612434 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.612543 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612449 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.612543 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.612466 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.613995 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.613964 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "kube-api-access-wwpfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:05.614149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.614093 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:49:05.614485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.614464 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:05.614485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.614480 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token" pod "7404d52a-793e-4bd1-a262-8ee7e49e0c8e" (UID: "7404d52a-793e-4bd1-a262-8ee7e49e0c8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:05.713663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.713629 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpfg\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-kube-api-access-wwpfg\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.713663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.713660 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.713663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.713672 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.713867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.713680 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.713867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.713688 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7404d52a-793e-4bd1-a262-8ee7e49e0c8e-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:49:05.741157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.740046 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d45657-27b5-4070-aadc-b93e94487605" path="/var/lib/kubelet/pods/67d45657-27b5-4070-aadc-b93e94487605/volumes" Oct 30 01:49:05.876971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.876931 6275 generic.go:356] "Generic (PLEG): container finished" podID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerID="3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2" exitCode=0 Oct 30 01:49:05.877874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.877851 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" Oct 30 01:49:05.878390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.878364 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" event={"ID":"7404d52a-793e-4bd1-a262-8ee7e49e0c8e","Type":"ContainerDied","Data":"3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2"} Oct 30 01:49:05.878978 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.878955 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh" event={"ID":"7404d52a-793e-4bd1-a262-8ee7e49e0c8e","Type":"ContainerDied","Data":"2b44edd38faed21804f2069c7524414d6acf4b158f3f09be3c0810983da75f67"} Oct 30 01:49:05.879163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.879101 6275 scope.go:122] "RemoveContainer" containerID="3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2" Oct 30 01:49:05.895573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.895548 6275 scope.go:122] "RemoveContainer" containerID="f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8" Oct 30 01:49:05.904761 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.904736 6275 scope.go:122] "RemoveContainer" containerID="3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2" Oct 30 01:49:05.905069 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:05.905048 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2\": container with ID starting with 3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2 not found: ID does not exist" containerID="3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2" Oct 30 01:49:05.905198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.905080 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2"} err="failed to get container status \"3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2\": rpc error: code = NotFound desc = could not find container \"3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2\": container with ID starting with 3d786dc15a299218d76fa93dab0d501f837d839f025c70f571c2c19756c097e2 not found: ID does not exist" Oct 30 01:49:05.905198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.905098 6275 scope.go:122] "RemoveContainer" containerID="f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8" Oct 30 01:49:05.905429 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:05.905406 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8\": container with ID starting with f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8 not found: ID does not exist" containerID="f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8" Oct 30 01:49:05.905494 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.905440 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8"} err="failed to get container status \"f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8\": rpc error: code = NotFound desc = could not find container \"f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8\": container with ID starting with f7699bcf9f5d5f672459bd2f3bf09cb99258689cc37485563b83d3267699cca8 not found: ID does not exist" Oct 30 01:49:05.938096 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.938065 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh"] Oct 30 01:49:05.948417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:05.948387 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-14b53b75-openshift-default-66869b97f5-djcbh"] Oct 30 01:49:06.750340 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750299 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8"] Oct 30 01:49:06.750832 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750812 6275 cpu_manager.go:414] "containerMap: removing container" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" Oct 30 01:49:06.750925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750834 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d45657-27b5-4070-aadc-b93e94487605" containerName="istio-proxy" Oct 30 01:49:06.750925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750857 6275 cpu_manager.go:414] "containerMap: removing container" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" Oct 30 01:49:06.750925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750864 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" Oct 30 01:49:06.750925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750904 6275 cpu_manager.go:414] "containerMap: removing container" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" Oct 30 01:49:06.750925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.750912 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" Oct 30 01:49:06.751174 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.751002 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" Oct 30 01:49:06.754150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.754102 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9"] Oct 30 01:49:06.754270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.754213 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.754764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.754742 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" containerName="istio-proxy" Oct 30 01:49:06.756674 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.756655 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:06.769300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.769281 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b93d51-dockercfg-l65xv\"" Oct 30 01:49:06.776249 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.776219 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b93d51\"" Oct 30 01:49:06.780437 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.780414 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8"] Oct 30 01:49:06.782023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.782003 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9"] Oct 30 01:49:06.830760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.830711 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.830936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.830837 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhgq\" (UniqueName: \"kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq\") pod \"payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.830936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.830872 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfvd\" (UniqueName: \"kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd\") pod \"payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:06.830936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.830906 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:06.932569 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.932538 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.933007 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.932604 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhgq\" (UniqueName: \"kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq\") pod \"payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.933007 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.932630 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfvd\" (UniqueName: \"kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd\") pod \"payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:06.933007 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.932680 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:06.933381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.933362 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.933490 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.933471 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:06.954635 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.954603 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhgq\" (UniqueName: \"kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq\") pod \"payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:06.954776 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:06.954657 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfvd\" (UniqueName: \"kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd\") pod \"payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:07.067565 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.067535 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:07.074172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.074144 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:07.380075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.379989 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8"] Oct 30 01:49:07.383620 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:07.383583 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdf999e_4537_4613_896b_7c9736e8136e.slice/crio-d76afe4f9722d65f7cb2f779775f2d815d13c062c81e22093a86393a92749ab3 WatchSource:0}: Error finding container d76afe4f9722d65f7cb2f779775f2d815d13c062c81e22093a86393a92749ab3: Status 404 returned error can't find the container with id d76afe4f9722d65f7cb2f779775f2d815d13c062c81e22093a86393a92749ab3 Oct 30 01:49:07.391809 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.391784 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9"] Oct 30 01:49:07.397108 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:07.397079 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd25775_c6d6_4dcd_b19e_5db7800b899c.slice/crio-0c7e8db7b806acfbd83d312e881df9ce17123540e5ccb97f3d3dfc986710fe17 WatchSource:0}: Error finding container 0c7e8db7b806acfbd83d312e881df9ce17123540e5ccb97f3d3dfc986710fe17: Status 404 returned error can't find the container with id 0c7e8db7b806acfbd83d312e881df9ce17123540e5ccb97f3d3dfc986710fe17 Oct 30 01:49:07.741588 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.740617 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7404d52a-793e-4bd1-a262-8ee7e49e0c8e" path="/var/lib/kubelet/pods/7404d52a-793e-4bd1-a262-8ee7e49e0c8e/volumes" Oct 30 01:49:07.890251 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.890210 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" event={"ID":"8cdf999e-4537-4613-896b-7c9736e8136e","Type":"ContainerStarted","Data":"a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333"} Oct 30 01:49:07.890417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.890275 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" event={"ID":"8cdf999e-4537-4613-896b-7c9736e8136e","Type":"ContainerStarted","Data":"d76afe4f9722d65f7cb2f779775f2d815d13c062c81e22093a86393a92749ab3"} Oct 30 01:49:07.890417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.890369 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:07.891937 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.891910 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" event={"ID":"fbd25775-c6d6-4dcd-b19e-5db7800b899c","Type":"ContainerStarted","Data":"f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473"} Oct 30 01:49:07.892057 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.891961 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" event={"ID":"fbd25775-c6d6-4dcd-b19e-5db7800b899c","Type":"ContainerStarted","Data":"0c7e8db7b806acfbd83d312e881df9ce17123540e5ccb97f3d3dfc986710fe17"} Oct 30 01:49:07.892057 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.891990 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:07.928775 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.928727 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" podStartSLOduration=1.928712769 podStartE2EDuration="1.928712769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:07.927677404 +0000 UTC m=+1176.319485037" watchObservedRunningTime="2026-10-30 01:49:07.928712769 +0000 UTC m=+1176.320520401" Oct 30 01:49:07.965851 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:07.965798 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" podStartSLOduration=1.9657838939999999 podStartE2EDuration="1.965783894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:07.964495315 +0000 UTC m=+1176.356302947" watchObservedRunningTime="2026-10-30 01:49:07.965783894 +0000 UTC m=+1176.357591526" Oct 30 01:49:09.572006 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.571958 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7"] Oct 30 01:49:09.575722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.575693 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.598613 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.598582 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b93d51-serving-cert\"" Oct 30 01:49:09.603983 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.603955 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7"] Oct 30 01:49:09.683789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.683753 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.683972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.683844 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.683972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.683896 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.684099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.684063 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzj7\" (UniqueName: \"kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.787698 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.787622 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.801256 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.801218 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.802027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.802002 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzj7\" (UniqueName: \"kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.803960 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.799087 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.804707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.804685 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.819709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.814718 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.819709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.818628 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.831402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.831314 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzj7\" (UniqueName: \"kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7\") pod \"maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:09.868410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.868371 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 01:49:09.888560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:09.888529 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:10.307822 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:10.307787 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7"] Oct 30 01:49:10.311666 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:10.311639 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69742b2d_68ad_4041_af3c_c80740eca721.slice/crio-d39acb5b98e2f555414baa050c7d0387a24258a9b01e280e317336594d145611 WatchSource:0}: Error finding container d39acb5b98e2f555414baa050c7d0387a24258a9b01e280e317336594d145611: Status 404 returned error can't find the container with id d39acb5b98e2f555414baa050c7d0387a24258a9b01e280e317336594d145611 Oct 30 01:49:10.884221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:10.884194 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 01:49:11.002628 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:11.002554 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" event={"ID":"69742b2d-68ad-4041-af3c-c80740eca721","Type":"ContainerStarted","Data":"68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23"} Oct 30 01:49:11.002628 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:11.002605 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" event={"ID":"69742b2d-68ad-4041-af3c-c80740eca721","Type":"ContainerStarted","Data":"d39acb5b98e2f555414baa050c7d0387a24258a9b01e280e317336594d145611"} Oct 30 01:49:11.002804 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:11.002658 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:11.050314 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:11.050256 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" podStartSLOduration=1.714101473 podStartE2EDuration="2.050235797s" totalImagesPullingTime="336.134324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:11.048174275 +0000 UTC m=+1179.439981929" watchObservedRunningTime="2026-10-30 01:49:11.050235797 +0000 UTC m=+1179.442043431" Oct 30 01:49:14.770170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.770078 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2"] Oct 30 01:49:14.773009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.772987 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:14.793834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.793809 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c40a89-serving-cert\"" Oct 30 01:49:14.829168 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.823025 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2"] Oct 30 01:49:14.913776 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.913739 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:14.914016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.913843 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlvt\" (UniqueName: \"kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:14.914016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.913883 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:14.914154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:14.914099 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.016245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.016201 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.016558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.016522 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.016686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.016602 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlvt\" (UniqueName: \"kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.016686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.016659 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.017558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.017527 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.020687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.020609 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.021034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.021010 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.045141 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.044725 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlvt\" (UniqueName: \"kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt\") pod \"maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.085797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.085766 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:15.397049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:15.396972 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2"] Oct 30 01:49:15.400632 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:49:15.400604 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbc2cea_aff0_4325_9302_3605211bca38.slice/crio-c4db29e51b7075b7863a27e67ff29932621605854b661293393f7b8cab414f1b WatchSource:0}: Error finding container c4db29e51b7075b7863a27e67ff29932621605854b661293393f7b8cab414f1b: Status 404 returned error can't find the container with id c4db29e51b7075b7863a27e67ff29932621605854b661293393f7b8cab414f1b Oct 30 01:49:16.035278 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:16.035245 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" event={"ID":"2cbc2cea-aff0-4325-9302-3605211bca38","Type":"ContainerStarted","Data":"dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537"} Oct 30 01:49:16.035651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:16.035296 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" event={"ID":"2cbc2cea-aff0-4325-9302-3605211bca38","Type":"ContainerStarted","Data":"c4db29e51b7075b7863a27e67ff29932621605854b661293393f7b8cab414f1b"} Oct 30 01:49:16.035651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:16.035368 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:16.089777 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:16.089721 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" podStartSLOduration=1.807837806 podStartE2EDuration="2.089706065s" totalImagesPullingTime="281.868259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:16.089552932 +0000 UTC m=+1184.481360566" watchObservedRunningTime="2026-10-30 01:49:16.089706065 +0000 UTC m=+1184.481513698" Oct 30 01:49:16.695778 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:16.695734 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:49:17.013211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:17.013176 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 01:49:18.900039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:18.900006 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 01:49:18.900496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:18.900446 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 01:49:22.099317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:22.099282 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 01:49:26.800419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:26.800380 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:49:26.800937 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:26.800452 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 01:49:27.145992 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:27.145959 6275 generic.go:356] "Generic (PLEG): container finished" podID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerID="cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330" exitCode=0 Oct 30 01:49:27.146260 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:27.146008 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" event={"ID":"6b15538e-985f-47e5-9ebf-f4762969d4b3","Type":"ContainerDied","Data":"cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330"} Oct 30 01:49:27.146652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:27.146634 6275 scope.go:122] "RemoveContainer" containerID="cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330" Oct 30 01:49:27.148049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:27.148021 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:27.148147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:27.148092 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:27.148194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:27.148143 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:28.178147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:28.174340 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" event={"ID":"6b15538e-985f-47e5-9ebf-f4762969d4b3","Type":"ContainerStarted","Data":"e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67"} Oct 30 01:49:28.557910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:28.557454 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:49:28.557910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:28.557589 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:49:28.557910 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:28.557633 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 01:49:29.181826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.181738 6275 generic.go:356] "Generic (PLEG): container finished" podID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerID="a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b" exitCode=0 Oct 30 01:49:29.181826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.181810 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" event={"ID":"6b68df8f-6283-4f8e-9e70-22ac355cc809","Type":"ContainerDied","Data":"a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b"} Oct 30 01:49:29.182398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.182279 6275 scope.go:122] "RemoveContainer" containerID="a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b" Oct 30 01:49:29.183489 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.183456 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:29.183604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.183542 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:29.183604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.183594 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:29.561015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.560508 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:49:29.561015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:29.560575 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 01:49:30.204291 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:30.204232 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" event={"ID":"6b68df8f-6283-4f8e-9e70-22ac355cc809","Type":"ContainerStarted","Data":"8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8"} Oct 30 01:49:30.557976 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:30.557513 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:49:30.557976 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:30.557578 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 01:49:30.744193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:30.743711 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:30.744193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:30.743845 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:30.744193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:30.743899 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:49:31.560946 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.560503 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:49:31.560946 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.560564 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 01:49:31.760147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.760015 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:31.760147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.760076 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:49:31.829269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.829182 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:31.829269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.829232 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:31.900574 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.900547 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:49:31.900721 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:31.900552 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:49:32.220038 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.219950 6275 generic.go:356] "Generic (PLEG): container finished" podID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerID="9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc" exitCode=0 Oct 30 01:49:32.220242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.220058 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" event={"ID":"d57544d9-f1dc-492f-aa75-d00704637f9b","Type":"ContainerDied","Data":"9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc"} Oct 30 01:49:32.220535 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.220514 6275 scope.go:122] "RemoveContainer" containerID="9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc" Oct 30 01:49:32.221894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.221864 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:32.221990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.221927 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:32.221990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.221957 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:49:32.559144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.558513 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:49:32.559144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.558579 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 01:49:32.758145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.757509 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:32.758145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:32.757568 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:49:33.228637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.228593 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" event={"ID":"d57544d9-f1dc-492f-aa75-d00704637f9b","Type":"ContainerStarted","Data":"accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0"} Oct 30 01:49:33.572143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.565460 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:33.572143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.565581 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:33.572143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.565625 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:33.746148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.742475 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:33.746148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.742535 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:49:33.859328 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:33.859233 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:34.149818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.148400 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:49:34.149818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.149619 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:49:34.211301 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.211265 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 01:49:34.560145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.559534 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:34.560145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.559599 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:34.746149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.743489 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:34.746149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:34.743552 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:49:35.561144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:35.560553 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:35.561144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:35.560620 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:35.815427 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:35.815323 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:35.817001 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:35.816975 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 01:49:36.564142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:36.563716 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:36.564142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:36.563790 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:37.560514 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:37.560479 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:37.560808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:37.560778 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:38.560567 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:38.560529 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:49:38.561176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:38.561142 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:49:39.563456 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:39.563411 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:39.563932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:39.563827 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:39.564729 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:49:39.564710 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 01:49:46.696182 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:49:46.696146 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:50:16.696151 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:50:16.696096 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:50:24.348644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.348609 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g"] Oct 30 01:50:24.352755 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.352736 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.368187 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.368165 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"kube-root-ca.crt\"" Oct 30 01:50:24.378676 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.378646 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"openshift-service-ca.crt\"" Oct 30 01:50:24.378810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.378732 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"default-dockercfg-rp6jt\"" Oct 30 01:50:24.380227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.380200 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"premium-w0-c40a89-kserve-self-signed-certs\"" Oct 30 01:50:24.381464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.381440 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g"] Oct 30 01:50:24.444935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.444903 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.445095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.444944 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmz6k\" (UniqueName: \"kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.445095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.445062 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.445207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.445111 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.445252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.445237 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.445289 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.445275 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.546723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.546683 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.546906 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.546763 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.548416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.547035 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.548416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.547289 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmz6k\" (UniqueName: \"kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.548416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.547445 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.548416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.547527 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.548416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.548050 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.549338 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.549037 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.549730 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.549691 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.551155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.551110 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.568907 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.568878 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.571036 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.571008 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmz6k\" (UniqueName: \"kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k\") pod \"premium-w0-c40a89-kserve-6d9c5b5899-ncd8g\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.664473 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.664390 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:24.967669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:24.967589 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g"] Oct 30 01:50:24.971313 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:50:24.971285 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaf72a41_7c0d_4851_9fc9_57de3e8a42d3.slice/crio-86c7f11c7e84f1ee237d715381f77c8427432069e79a7a852599b1e0624fccc1 WatchSource:0}: Error finding container 86c7f11c7e84f1ee237d715381f77c8427432069e79a7a852599b1e0624fccc1: Status 404 returned error can't find the container with id 86c7f11c7e84f1ee237d715381f77c8427432069e79a7a852599b1e0624fccc1 Oct 30 01:50:25.596986 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:25.596950 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" event={"ID":"daf72a41-7c0d-4851-9fc9-57de3e8a42d3","Type":"ContainerStarted","Data":"624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473"} Oct 30 01:50:25.597395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:25.597001 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" event={"ID":"daf72a41-7c0d-4851-9fc9-57de3e8a42d3","Type":"ContainerStarted","Data":"86c7f11c7e84f1ee237d715381f77c8427432069e79a7a852599b1e0624fccc1"} Oct 30 01:50:25.636405 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:25.636349 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podStartSLOduration=1.636332172 podStartE2EDuration="1.636332172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:25.634810511 +0000 UTC m=+1254.026618144" watchObservedRunningTime="2026-10-30 01:50:25.636332172 +0000 UTC m=+1254.028139858" Oct 30 01:50:25.665037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:25.665010 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:34.664876 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:34.664840 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:34.679687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:34.679660 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:35.664026 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:50:35.663993 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:50:46.705031 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:50:46.704950 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:51:05.729192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.729156 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2"] Oct 30 01:51:05.736688 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.736656 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.760240 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.760216 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f6f4f0-openshift-default-dockercfg-jl28w\"" Oct 30 01:51:05.766443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.766416 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2"] Oct 30 01:51:05.866841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.866804 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.866841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.866845 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.866908 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.866934 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.867056 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.867076 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.867103 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867241 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.867188 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzqb\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.867241 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.867213 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968252 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968286 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968312 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968354 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzqb\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968385 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968461 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968489 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968730 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968705 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968815 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968797 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968915 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968894 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.968983 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968945 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.969035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.968994 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.969035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.969012 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.969158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.969082 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.970704 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.970684 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.970924 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.970904 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.989816 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.989754 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzqb\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:05.989912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:05.989895 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:06.050618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.050586 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:06.365341 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.365308 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2"] Oct 30 01:51:06.368370 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:51:06.368343 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a1531b_aeaa_4143_af50_e4b65d5f3d20.slice/crio-8a20938dd5aaa34d0e69d94d8fe57ea9308bf66bd939296c2bd51fc981abb79c WatchSource:0}: Error finding container 8a20938dd5aaa34d0e69d94d8fe57ea9308bf66bd939296c2bd51fc981abb79c: Status 404 returned error can't find the container with id 8a20938dd5aaa34d0e69d94d8fe57ea9308bf66bd939296c2bd51fc981abb79c Oct 30 01:51:06.370565 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.370532 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:06.370644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.370601 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:06.370644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.370630 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:06.844339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.843196 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" event={"ID":"01a1531b-aeaa-4143-af50-e4b65d5f3d20","Type":"ContainerStarted","Data":"3c35cf5c8f66f6ed6a66e3d1afc86a346755643a9ef217fc4e69aee8a045f716"} Oct 30 01:51:06.844339 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.843265 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" event={"ID":"01a1531b-aeaa-4143-af50-e4b65d5f3d20","Type":"ContainerStarted","Data":"8a20938dd5aaa34d0e69d94d8fe57ea9308bf66bd939296c2bd51fc981abb79c"} Oct 30 01:51:06.910338 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:06.910285 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podStartSLOduration=1.91027176 podStartE2EDuration="1.91027176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:06.898922268 +0000 UTC m=+1295.290729901" watchObservedRunningTime="2026-10-30 01:51:06.91027176 +0000 UTC m=+1295.302079392" Oct 30 01:51:07.056145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:07.053350 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:07.060143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:07.059849 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:51:07.060143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:07.059917 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:51:08.061147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:08.058514 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:51:08.061147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:08.058578 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:51:09.058150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:09.054537 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:51:09.058150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:09.054606 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:51:10.058146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.053509 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:51:10.058146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.053571 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:51:10.490722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.490632 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g"] Oct 30 01:51:10.494808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.494787 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.510636 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.510611 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-78d5ff-openshift-default-dockercfg-s66rj\"" Oct 30 01:51:10.526273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526223 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526323 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkkv\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526424 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526507 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526479 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526631 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526609 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526688 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526666 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526720 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526822 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526807 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.526871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.526841 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.527547 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.527521 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g"] Oct 30 01:51:10.628237 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628205 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628250 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628294 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628320 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628335 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628392 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628421 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628488 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628510 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkkv\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628755 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.628965 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.628939 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.629112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.629015 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.629112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.629083 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.629290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.629270 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.630740 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.630720 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.631406 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.631388 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.666863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.666805 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkkv\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.667191 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.667166 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:10.808013 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:10.807964 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:11.064952 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.064663 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:51:11.064952 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.064738 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:51:11.170418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.166704 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g"] Oct 30 01:51:11.170739 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:51:11.170700 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03ff625_40fc_4827_a208_b72a2c83bcfe.slice/crio-a946284a6de394996d42643e6fe30f9ac00055fb359f816df627a72629f2480b WatchSource:0}: Error finding container a946284a6de394996d42643e6fe30f9ac00055fb359f816df627a72629f2480b: Status 404 returned error can't find the container with id a946284a6de394996d42643e6fe30f9ac00055fb359f816df627a72629f2480b Oct 30 01:51:11.175196 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.175156 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:11.175316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.175279 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:11.175383 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.175359 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:11.896792 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.896737 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" event={"ID":"f03ff625-40fc-4827-a208-b72a2c83bcfe","Type":"ContainerStarted","Data":"dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c"} Oct 30 01:51:11.896979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.896834 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" event={"ID":"f03ff625-40fc-4827-a208-b72a2c83bcfe","Type":"ContainerStarted","Data":"a946284a6de394996d42643e6fe30f9ac00055fb359f816df627a72629f2480b"} Oct 30 01:51:11.979604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:11.979539 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podStartSLOduration=1.979518415 podStartE2EDuration="1.979518415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:11.956582649 +0000 UTC m=+1300.348390299" watchObservedRunningTime="2026-10-30 01:51:11.979518415 +0000 UTC m=+1300.371326049" Oct 30 01:51:12.061145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:12.060298 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:51:12.061145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:12.060360 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:51:12.808899 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:12.808863 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:12.810458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:12.810436 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:51:12.810538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:12.810483 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:51:13.186798 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:13.186669 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:13.187541 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:13.187511 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:13.189538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:13.189515 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 01:51:13.808556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:13.808517 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:51:13.808726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:13.808571 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:51:14.809311 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:14.809269 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:51:14.809683 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:14.809330 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:51:15.808481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:15.808444 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:51:15.808678 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:15.808498 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:51:16.707049 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:51:16.707016 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:51:16.810108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:16.808380 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:51:16.810108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:16.808443 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:51:18.094062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:18.093986 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:18.097465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:18.097431 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:18.098614 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:18.098585 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:51:40.195734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.195696 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8"] Oct 30 01:51:40.201478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.201449 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.217381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.217349 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8"] Oct 30 01:51:40.225728 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.225701 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"distinct-w0-c40a89-kserve-self-signed-certs\"" Oct 30 01:51:40.258707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.258671 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcc5m\" (UniqueName: \"kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.258859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.258780 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.258859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.258807 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.258859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.258823 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.258859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.258859 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.258996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.258916 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360377 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcc5m\" (UniqueName: \"kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360444 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360463 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360477 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360507 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360523 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.360908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360881 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.361004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.360980 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.361077 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.361025 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.362873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.362852 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.363047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.363030 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.391579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.391511 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcc5m\" (UniqueName: \"kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m\") pod \"distinct-w0-c40a89-kserve-6d56fb7698-dn7c8\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.513969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.513930 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:40.813017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:40.812930 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8"] Oct 30 01:51:40.816322 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:51:40.816293 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde2f5ccc_398f_443a_a4b4_891bbaae9b77.slice/crio-1b26613c34a49c5674af69d30433074b7abf9c1ace65fd3e26d429f02ae9a662 WatchSource:0}: Error finding container 1b26613c34a49c5674af69d30433074b7abf9c1ace65fd3e26d429f02ae9a662: Status 404 returned error can't find the container with id 1b26613c34a49c5674af69d30433074b7abf9c1ace65fd3e26d429f02ae9a662 Oct 30 01:51:41.095624 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:41.095545 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" event={"ID":"de2f5ccc-398f-443a-a4b4-891bbaae9b77","Type":"ContainerStarted","Data":"55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0"} Oct 30 01:51:41.095624 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:41.095584 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" event={"ID":"de2f5ccc-398f-443a-a4b4-891bbaae9b77","Type":"ContainerStarted","Data":"1b26613c34a49c5674af69d30433074b7abf9c1ace65fd3e26d429f02ae9a662"} Oct 30 01:51:41.136132 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:41.136061 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podStartSLOduration=1.136043237 podStartE2EDuration="1.136043237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:41.133959815 +0000 UTC m=+1329.525767447" watchObservedRunningTime="2026-10-30 01:51:41.136043237 +0000 UTC m=+1329.527850871" Oct 30 01:51:41.514852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:41.514825 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:46.700452 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:51:46.700409 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:51:50.514403 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:50.514364 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:50.526642 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:50.526616 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:51.165857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:51.165831 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:51:54.791252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.791209 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr"] Oct 30 01:51:54.794928 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.794907 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.818168 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.818109 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr"] Oct 30 01:51:54.933741 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.933707 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.933741 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.933749 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.933981 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.933866 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.933981 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.933929 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmt7l\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.933981 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.933959 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.934088 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.934012 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.934195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.934180 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.934249 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.934237 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:54.934332 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:54.934311 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035056 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035019 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035088 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035154 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035190 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035223 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035291 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035372 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035350 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmt7l\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035381 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035445 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035819 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035713 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.035913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.035889 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.036021 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.036000 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.036092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.036071 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.036344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.036324 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.037803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.037780 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.037906 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.037821 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.056985 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.056931 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.057084 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.057005 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmt7l\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.108805 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.108778 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:55.427882 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.427847 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr"] Oct 30 01:51:55.430901 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:51:55.430870 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2859bee_9ad3_4267_96fb_7ea985b726be.slice/crio-790870cbafff561a0c8884a3302cc59471c334d580eb8508bc00009841081b38 WatchSource:0}: Error finding container 790870cbafff561a0c8884a3302cc59471c334d580eb8508bc00009841081b38: Status 404 returned error can't find the container with id 790870cbafff561a0c8884a3302cc59471c334d580eb8508bc00009841081b38 Oct 30 01:51:55.433210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.433174 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:55.433342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.433253 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:55.433342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:55.433320 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:51:56.191198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:56.191166 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" event={"ID":"b2859bee-9ad3-4267-96fb-7ea985b726be","Type":"ContainerStarted","Data":"d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979"} Oct 30 01:51:56.191623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:56.191225 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" event={"ID":"b2859bee-9ad3-4267-96fb-7ea985b726be","Type":"ContainerStarted","Data":"790870cbafff561a0c8884a3302cc59471c334d580eb8508bc00009841081b38"} Oct 30 01:51:56.263189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:56.257768 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podStartSLOduration=2.257749528 podStartE2EDuration="2.257749528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:56.256694781 +0000 UTC m=+1344.648502414" watchObservedRunningTime="2026-10-30 01:51:56.257749528 +0000 UTC m=+1344.649557162" Oct 30 01:51:57.109008 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:57.108975 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:51:57.110611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:57.110581 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:51:57.110744 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:57.110633 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:51:58.109340 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:58.109302 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:51:58.109796 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:58.109361 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:51:59.109634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:59.109599 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:51:59.110042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:51:59.109657 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:52:00.109869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:00.109830 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:52:00.110330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:00.109891 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:52:01.109478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:01.109442 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:52:01.109694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:01.109506 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:52:02.188784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:02.188753 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:52:02.189411 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:02.189382 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:52:02.190648 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:02.190619 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 01:52:05.914079 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:05.914040 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct"] Oct 30 01:52:05.918422 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:05.918399 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:05.942567 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:05.942534 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct"] Oct 30 01:52:06.083979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.083943 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.083987 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084009 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2p7r\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084107 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084233 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084264 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084325 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084361 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.084591 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.084395 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.185896 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.185800 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.185896 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.185864 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.185927 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.185964 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186019 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186074 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186107 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186404 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186156 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2p7r\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186404 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186193 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186507 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186462 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.186559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.186504 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.187035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.187015 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.187114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.187086 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.187209 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.187108 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.188801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.188772 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.189086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.189066 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.207953 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.207928 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.208349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.208329 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2p7r\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.234464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.234437 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:06.552599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.552564 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct"] Oct 30 01:52:06.556363 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:52:06.556331 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c69cb42_d10c_41c5_9529_441098f555f4.slice/crio-947b085dbe254a04be1c0812c94711e0c8732389bb80a8b9acd68a25a899a51c WatchSource:0}: Error finding container 947b085dbe254a04be1c0812c94711e0c8732389bb80a8b9acd68a25a899a51c: Status 404 returned error can't find the container with id 947b085dbe254a04be1c0812c94711e0c8732389bb80a8b9acd68a25a899a51c Oct 30 01:52:06.559202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.559160 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:06.559311 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.559258 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:06.559311 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:06.559298 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:07.271154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:07.270841 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" event={"ID":"2c69cb42-d10c-41c5-9529-441098f555f4","Type":"ContainerStarted","Data":"4aca65518fb2c38d3133332b59aac24a8f9510deefbf92ff9a05649809e265a2"} Oct 30 01:52:07.271154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:07.270953 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" event={"ID":"2c69cb42-d10c-41c5-9529-441098f555f4","Type":"ContainerStarted","Data":"947b085dbe254a04be1c0812c94711e0c8732389bb80a8b9acd68a25a899a51c"} Oct 30 01:52:07.326244 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:07.326186 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podStartSLOduration=2.326169894 podStartE2EDuration="2.326169894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:07.324006089 +0000 UTC m=+1355.715813720" watchObservedRunningTime="2026-10-30 01:52:07.326169894 +0000 UTC m=+1355.717977527" Oct 30 01:52:08.235657 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:08.235622 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:08.236867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:08.236838 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 01:52:08.237012 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:08.236892 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 01:52:09.234901 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:09.234862 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 01:52:09.235338 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:09.234927 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 01:52:10.235486 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:10.235452 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 01:52:10.235893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:10.235508 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 01:52:11.235449 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:11.235415 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 01:52:11.235706 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:11.235469 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 01:52:12.234974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:12.234894 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 01:52:12.234974 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:12.234957 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 01:52:13.548934 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:13.548892 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:13.550030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:13.550003 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:13.550912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:13.550894 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 01:52:16.728855 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:52:16.728812 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:52:20.142063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.142031 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm"] Oct 30 01:52:20.146226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.146204 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.170011 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.169982 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"default-dockercfg-hv9nq\"" Oct 30 01:52:20.170151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.170019 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"openshift-service-ca.crt\"" Oct 30 01:52:20.170151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.170052 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"distinct-2-w1-f6f4f0-kserve-self-signed-certs\"" Oct 30 01:52:20.170151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.170101 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"kube-root-ca.crt\"" Oct 30 01:52:20.172854 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.172832 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm"] Oct 30 01:52:20.257217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.257182 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.257387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.257260 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.257387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.257319 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.257465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.257394 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.257465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.257443 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.257533 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.257519 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwgz\" (UniqueName: \"kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.362935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.362886 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.363177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.362964 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.363177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.363082 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.363177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.363160 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.363399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.363220 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwgz\" (UniqueName: \"kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.363399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.363343 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.365073 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.363989 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.365073 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.364330 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.365073 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.364962 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.388189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.384765 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.388189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.385342 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.411881 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.411789 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwgz\" (UniqueName: \"kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz\") pod \"distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.464362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.464324 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:20.760889 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:20.760724 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm"] Oct 30 01:52:20.763734 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:52:20.763703 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c431280_4854_4765_8c94_9cee52957245.slice/crio-80378f94c0c9291ffca7e5b84fba6f69d6405857ba0be92f48c8eef32d4de475 WatchSource:0}: Error finding container 80378f94c0c9291ffca7e5b84fba6f69d6405857ba0be92f48c8eef32d4de475: Status 404 returned error can't find the container with id 80378f94c0c9291ffca7e5b84fba6f69d6405857ba0be92f48c8eef32d4de475 Oct 30 01:52:21.426908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:21.426868 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" event={"ID":"2c431280-4854-4765-8c94-9cee52957245","Type":"ContainerStarted","Data":"ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75"} Oct 30 01:52:21.427327 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:21.426918 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" event={"ID":"2c431280-4854-4765-8c94-9cee52957245","Type":"ContainerStarted","Data":"80378f94c0c9291ffca7e5b84fba6f69d6405857ba0be92f48c8eef32d4de475"} Oct 30 01:52:21.465240 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:21.465203 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:21.466436 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:21.466394 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podStartSLOduration=1.466381044 podStartE2EDuration="1.466381044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:21.465561348 +0000 UTC m=+1369.857368983" watchObservedRunningTime="2026-10-30 01:52:21.466381044 +0000 UTC m=+1369.858188679" Oct 30 01:52:25.726732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.726698 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4"] Oct 30 01:52:25.731138 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.731100 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.758560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.758526 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4"] Oct 30 01:52:25.822747 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.822718 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.822928 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.822756 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.822928 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.822853 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.823050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.823011 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.823108 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.823080 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.823204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.823174 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2t6\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.823261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.823225 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.823357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.823333 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.823416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.823367 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924235 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924196 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924405 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924315 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924405 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924358 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924457 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924527 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924512 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924565 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924632 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924610 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2t6\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924651 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924743 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924699 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924743 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924723 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.924855 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924749 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.925004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924967 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.925004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.924994 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.925199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.925081 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.926954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.926936 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.927093 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.927076 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.946307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.946278 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:25.946508 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:25.946487 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2t6\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:26.045914 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.045884 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:26.362053 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.361966 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4"] Oct 30 01:52:26.365372 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:52:26.365342 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2143f36d_f231_4f4c_bd77_84fd468cfb15.slice/crio-ef83008a73fbfa803689b5829cc6cbb5fac73cb0feaff80b181e6e13d75cd7a4 WatchSource:0}: Error finding container ef83008a73fbfa803689b5829cc6cbb5fac73cb0feaff80b181e6e13d75cd7a4: Status 404 returned error can't find the container with id ef83008a73fbfa803689b5829cc6cbb5fac73cb0feaff80b181e6e13d75cd7a4 Oct 30 01:52:26.367629 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.367599 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:26.367707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.367667 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:26.367707 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.367695 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:26.457544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.457507 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" event={"ID":"2143f36d-f231-4f4c-bd77-84fd468cfb15","Type":"ContainerStarted","Data":"fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f"} Oct 30 01:52:26.457745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.457560 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" event={"ID":"2143f36d-f231-4f4c-bd77-84fd468cfb15","Type":"ContainerStarted","Data":"ef83008a73fbfa803689b5829cc6cbb5fac73cb0feaff80b181e6e13d75cd7a4"} Oct 30 01:52:26.501598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:26.501532 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podStartSLOduration=1.501515632 podStartE2EDuration="1.501515632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:26.499691461 +0000 UTC m=+1374.891499109" watchObservedRunningTime="2026-10-30 01:52:26.501515632 +0000 UTC m=+1374.893323265" Oct 30 01:52:27.047357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:27.046442 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:27.048631 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:27.048597 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:27.048879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:27.048842 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:28.063692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:28.063652 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:28.064378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:28.064338 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:29.048572 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:29.048531 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:29.048896 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:29.048867 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:30.059147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:30.050629 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:30.059147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:30.050729 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:30.464919 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:30.464822 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:30.477688 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:30.477661 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:30.508570 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:30.508543 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:52:31.049158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:31.048832 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:31.049158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:31.048893 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:32.051551 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.051512 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:32.052516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.052099 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:32.067232 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.064646 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z"] Oct 30 01:52:32.070705 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.070676 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.091721 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.091692 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z"] Oct 30 01:52:32.175796 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.175755 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.175948 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.175995 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8dn\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.176059 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.176103 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.176149 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.176188 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.176256 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.176323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.176279 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277476 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277513 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277561 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277659 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277693 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8dn\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277755 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277796 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277822 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.277863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277857 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.278053 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.277933 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.278268 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.278237 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.278268 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.278253 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.278407 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.278296 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.278644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.278616 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.280610 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.280588 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.280758 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.280740 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.299456 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.299424 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.299557 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.299461 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8dn\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.386757 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.386669 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:32.708571 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.708492 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z"] Oct 30 01:52:32.711886 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:52:32.711850 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314bc821_be39_4f39_9fae_949d1c952dd0.slice/crio-0c6f55618e235aa42b71c58b4c41b1fee2d4f7ce91d78c736e78ad0abe55e665 WatchSource:0}: Error finding container 0c6f55618e235aa42b71c58b4c41b1fee2d4f7ce91d78c736e78ad0abe55e665: Status 404 returned error can't find the container with id 0c6f55618e235aa42b71c58b4c41b1fee2d4f7ce91d78c736e78ad0abe55e665 Oct 30 01:52:32.714663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.714617 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:32.714739 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.714720 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:32.714790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:32.714772 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:33.047261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:33.047220 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:52:33.047439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:33.047293 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:52:33.518847 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:33.518803 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" event={"ID":"314bc821-be39-4f39-9fae-949d1c952dd0","Type":"ContainerStarted","Data":"2be506590686af71806082a6f1e41e36cfd4231a387f86d828954989ab79efbe"} Oct 30 01:52:33.519262 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:33.518878 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" event={"ID":"314bc821-be39-4f39-9fae-949d1c952dd0","Type":"ContainerStarted","Data":"0c6f55618e235aa42b71c58b4c41b1fee2d4f7ce91d78c736e78ad0abe55e665"} Oct 30 01:52:33.587076 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:33.587010 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podStartSLOduration=1.586993932 podStartE2EDuration="1.586993932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:33.585686526 +0000 UTC m=+1381.977494159" watchObservedRunningTime="2026-10-30 01:52:33.586993932 +0000 UTC m=+1381.978801566" Oct 30 01:52:34.072477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:34.071203 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:34.072477 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:34.072420 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:34.072901 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:34.072881 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:52:34.387502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:34.387423 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:34.388894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:34.388867 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:52:34.389022 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:34.388914 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:52:35.387418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:35.387383 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:52:35.387851 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:35.387438 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:52:36.388009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:36.387970 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:52:36.388531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:36.388044 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:52:37.211198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.211161 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt"] Oct 30 01:52:37.215455 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.215431 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.269364 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.269327 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"default-dockercfg-wr9nk\"" Oct 30 01:52:37.269364 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.269327 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"kube-root-ca.crt\"" Oct 30 01:52:37.269642 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.269402 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"distinct-2-w2-719029-kserve-self-signed-certs\"" Oct 30 01:52:37.269642 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.269428 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"openshift-service-ca.crt\"" Oct 30 01:52:37.273256 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.273226 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt"] Oct 30 01:52:37.286073 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.286037 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.286239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.286089 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.286239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.286220 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.286344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.286251 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.286344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.286286 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.286430 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.286405 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mxn\" (UniqueName: \"kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.388954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.388910 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.405412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.405383 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.416483 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.389822 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:52:37.416852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.409541 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.418603 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.418578 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.418800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.418775 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:52:37.427399 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.427368 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.428020 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.427674 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mxn\" (UniqueName: \"kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.428427 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.428384 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.429267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.429248 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.429474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.429351 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.429617 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.429597 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.433947 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.433921 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.467324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.467201 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mxn\" (UniqueName: \"kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn\") pod \"distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.529637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.529606 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:37.866783 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:37.866754 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt"] Oct 30 01:52:37.870276 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:52:37.870249 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3b7b172_34fb_461b_ba7a_ae4a5c994600.slice/crio-09af46c3f50fccd8b1d42e79d2f510d5efda093448f5952d0c057e72108fa147 WatchSource:0}: Error finding container 09af46c3f50fccd8b1d42e79d2f510d5efda093448f5952d0c057e72108fa147: Status 404 returned error can't find the container with id 09af46c3f50fccd8b1d42e79d2f510d5efda093448f5952d0c057e72108fa147 Oct 30 01:52:38.387715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:38.387679 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:52:38.387903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:38.387739 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:52:38.576681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:38.576648 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" event={"ID":"c3b7b172-34fb-461b-ba7a-ae4a5c994600","Type":"ContainerStarted","Data":"6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696"} Oct 30 01:52:38.577063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:38.576689 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" event={"ID":"c3b7b172-34fb-461b-ba7a-ae4a5c994600","Type":"ContainerStarted","Data":"09af46c3f50fccd8b1d42e79d2f510d5efda093448f5952d0c057e72108fa147"} Oct 30 01:52:38.615665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:38.615618 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podStartSLOduration=1.615606254 podStartE2EDuration="1.615606254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:38.615281012 +0000 UTC m=+1387.007088646" watchObservedRunningTime="2026-10-30 01:52:38.615606254 +0000 UTC m=+1387.007413886" Oct 30 01:52:39.387285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:39.387249 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:52:39.387453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:39.387314 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:52:39.530022 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:39.529989 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:40.392243 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:40.392212 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:40.392648 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:40.392626 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:40.393463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:40.393447 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:52:46.696886 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:52:46.696850 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:52:47.530104 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:47.530076 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:47.542609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:47.542584 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:47.648680 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:47.648650 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:52:50.505158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:50.505093 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 01:52:50.642296 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:50.642236 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:52:50.675369 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:50.675323 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:52:51.062297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.062134 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn"] Oct 30 01:52:51.066708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.066633 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.105192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.105157 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn"] Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129642 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129696 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwgh\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129750 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129807 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129831 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129915 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.129970 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.130033 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.130148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.130066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.250516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.250471 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.250834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.250805 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.251037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.250998 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.251250 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.251217 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwgh\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.259886 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.251575 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.260398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.260365 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.260854 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.260834 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.261270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.261251 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.261856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.261828 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.262089 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.262070 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.262446 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.262420 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.263075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.263053 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.268431 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.268405 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.268881 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.268859 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.309002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.308968 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.309530 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.309504 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.316426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.316355 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwgh\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.316799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.316776 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.407163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.407108 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:51.728763 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.728688 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn"] Oct 30 01:52:51.732015 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:52:51.731984 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5421aa_5367_446c_b3f2_8261bf43d585.slice/crio-578cb744632f68a7a1fb4f0b4e3513117ead8d26ee0c3f6906b4736039ce7d6d WatchSource:0}: Error finding container 578cb744632f68a7a1fb4f0b4e3513117ead8d26ee0c3f6906b4736039ce7d6d: Status 404 returned error can't find the container with id 578cb744632f68a7a1fb4f0b4e3513117ead8d26ee0c3f6906b4736039ce7d6d Oct 30 01:52:51.734378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.734346 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:51.734481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.734402 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:51.734481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:51.734434 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:52:52.698301 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:52.698269 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" event={"ID":"ae5421aa-5367-446c-b3f2-8261bf43d585","Type":"ContainerStarted","Data":"6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70"} Oct 30 01:52:52.698480 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:52.698320 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" event={"ID":"ae5421aa-5367-446c-b3f2-8261bf43d585","Type":"ContainerStarted","Data":"578cb744632f68a7a1fb4f0b4e3513117ead8d26ee0c3f6906b4736039ce7d6d"} Oct 30 01:52:52.763149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:52.762267 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podStartSLOduration=1.762246617 podStartE2EDuration="1.762246617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:52.761467459 +0000 UTC m=+1401.153275093" watchObservedRunningTime="2026-10-30 01:52:52.762246617 +0000 UTC m=+1401.154054251" Oct 30 01:52:53.407593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:53.407558 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:53.409546 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:53.409523 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:52:53.409645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:53.409571 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:52:54.408511 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:54.408470 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:52:54.409008 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:54.408534 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:52:55.408221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:55.408187 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:52:55.408404 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:55.408242 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:52:56.407883 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:56.407835 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:52:56.408323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:56.407905 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:52:57.407915 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:57.407879 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:52:57.408290 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:57.407941 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:52:58.408501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:58.408465 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:52:58.408893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:58.408529 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:52:59.412843 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:59.412811 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:59.413359 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:59.413211 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:52:59.414161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:52:59.414114 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 01:53:02.138719 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.138680 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4"] Oct 30 01:53:02.143919 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.143895 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.176401 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.176364 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4"] Oct 30 01:53:02.232441 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232397 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcs2\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.232634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232505 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.232634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232561 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.232634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232592 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.232823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232626 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.232823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232776 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.232931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.232911 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.233024 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.233009 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.233095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.233082 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334313 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcs2\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334524 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334454 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334524 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334509 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334543 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334574 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334593 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334649 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334774 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334755 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334832 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.334959 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334937 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.335023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.334945 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.335286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.335261 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.335387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.335356 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.335439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.335410 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.337053 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.337035 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.337391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.337373 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.356012 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.355985 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.356401 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.356379 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcs2\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.462787 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.462712 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:02.792176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.792109 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4"] Oct 30 01:53:02.794769 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:53:02.794743 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6b46a2_a560_487b_b2f9_c456278bb868.slice/crio-053232f981cf0da15167bafbac8fe1c5fed623347b5bcb0a6e15138a5d93d6fe WatchSource:0}: Error finding container 053232f981cf0da15167bafbac8fe1c5fed623347b5bcb0a6e15138a5d93d6fe: Status 404 returned error can't find the container with id 053232f981cf0da15167bafbac8fe1c5fed623347b5bcb0a6e15138a5d93d6fe Oct 30 01:53:02.797037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.797005 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:02.797113 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.797069 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:02.797113 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.797098 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:02.815853 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:02.815821 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" event={"ID":"bd6b46a2-a560-487b-b2f9-c456278bb868","Type":"ContainerStarted","Data":"053232f981cf0da15167bafbac8fe1c5fed623347b5bcb0a6e15138a5d93d6fe"} Oct 30 01:53:03.823296 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:03.823258 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" event={"ID":"bd6b46a2-a560-487b-b2f9-c456278bb868","Type":"ContainerStarted","Data":"5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2"} Oct 30 01:53:03.890225 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:03.890169 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podStartSLOduration=1.8901515660000001 podStartE2EDuration="1.890151566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:03.889464469 +0000 UTC m=+1412.281272156" watchObservedRunningTime="2026-10-30 01:53:03.890151566 +0000 UTC m=+1412.281959197" Oct 30 01:53:04.463352 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:04.463314 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:04.465165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:04.465096 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:04.465293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:04.465161 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:53:05.463560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:05.463524 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:05.464030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:05.463644 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:53:06.463607 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:06.463563 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:06.464172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:06.463618 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:53:07.465146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:07.464552 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:07.465146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:07.464625 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:53:08.463623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:08.463582 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:08.463801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:08.463653 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:53:09.463461 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:09.463423 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:09.463879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:09.463496 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:53:10.014904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.014864 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49"] Oct 30 01:53:10.020521 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.020498 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.043204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.043176 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49"] Oct 30 01:53:10.115713 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.115680 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.115891 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.115735 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.115891 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.115832 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.116051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.115891 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.116051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.115992 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.116051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.116025 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.116176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.116052 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5lpq\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.116211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.116173 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.116326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.116297 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217244 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217285 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217302 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5lpq\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217343 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217409 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217493 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217527 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217634 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217685 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217834 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217807 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217825 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.217958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.217876 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.218103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.218075 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.218430 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.218413 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.219752 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.219731 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.219897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.219881 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.240867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.240838 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.241347 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.241329 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5lpq\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.335381 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.335301 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:10.468257 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.468225 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:10.468712 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.468641 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:10.469497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.469475 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 01:53:10.655545 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.655434 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49"] Oct 30 01:53:10.658277 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:53:10.658252 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e16a09_5127_4940_b275_4f0d20250709.slice/crio-d35a36badec591e5f8429a0328e73bec89614505a4bff2884567618a582e44c7 WatchSource:0}: Error finding container d35a36badec591e5f8429a0328e73bec89614505a4bff2884567618a582e44c7: Status 404 returned error can't find the container with id d35a36badec591e5f8429a0328e73bec89614505a4bff2884567618a582e44c7 Oct 30 01:53:10.660696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.660655 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:10.660781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.660744 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:10.660818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.660787 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:10.925374 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.925278 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" event={"ID":"19e16a09-5127-4940-b275-4f0d20250709","Type":"ContainerStarted","Data":"01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0"} Oct 30 01:53:10.925374 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.925339 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" event={"ID":"19e16a09-5127-4940-b275-4f0d20250709","Type":"ContainerStarted","Data":"d35a36badec591e5f8429a0328e73bec89614505a4bff2884567618a582e44c7"} Oct 30 01:53:10.975035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:10.974851 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podStartSLOduration=1.974830983 podStartE2EDuration="1.974830983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:10.973790688 +0000 UTC m=+1419.365598321" watchObservedRunningTime="2026-10-30 01:53:10.974830983 +0000 UTC m=+1419.366638617" Oct 30 01:53:11.335691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:11.335649 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:11.337458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:11.337430 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:11.337604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:11.337490 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:12.336521 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:12.336478 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:12.336932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:12.336561 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:13.340144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:13.339526 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:13.340144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:13.339593 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:14.227146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:14.226226 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.340374 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:14.340243 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:14.341469 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:14.340907 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:14.506208 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:14.505904 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.652789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:14.651277 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.652789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:14.652098 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:53:15.233147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.231599 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.233147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.232365 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:53:15.512252 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.509639 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.535708 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.535672 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:53:15.592146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.590230 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:15.592146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.590300 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.657147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.656520 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:53:15.657147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.656967 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:53:15.672216 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.665593 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:15.672216 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.665641 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:15.708177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.679687 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.721341 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.708517 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:53:15.721341 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:15.709452 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 01:53:16.174148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:16.169554 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:53:16.346023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:16.345983 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:16.361391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:16.361343 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:16.526190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:16.520540 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.675682 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:16.675643 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.695165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:16.683490 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:17.061468 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:53:17.061428 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:53:17.076531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.073469 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:53:17.215182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.187744 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:17.255368 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.255335 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g"] Oct 30 01:53:17.341818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.341731 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:17.360275 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.360223 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:17.362217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.362191 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g"] Oct 30 01:53:17.362753 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.362734 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:53:17.363779 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.362638 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.531918 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.521990 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.532906 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533104 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533186 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533254 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533499 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533582 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzppb\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.533736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533625 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.534242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.533934 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.605549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.605448 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:53:17.639284 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.639238 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzppb\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.639964 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.639923 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.640723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.640700 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.640957 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.640938 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.641184 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.641163 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.641343 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.641326 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.641472 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.641455 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.641618 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.641603 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.641793 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.641777 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.642900 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.640554 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.644173 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.644149 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.644962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.644939 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.645370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.645348 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.646530 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.646506 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.686972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.686896 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.687771 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.687742 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.688783 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.688759 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.689607 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.689586 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzppb\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:17.805577 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:17.805547 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:18.168641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.168543 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g"] Oct 30 01:53:18.179338 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:53:18.179292 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7ecb81_e269_40d7_a976_b548e71d50b9.slice/crio-9b3ae207a4d9d88cc30244746f0a4dcf2212ad88fd6b2a553ecb62dfbeeedc68 WatchSource:0}: Error finding container 9b3ae207a4d9d88cc30244746f0a4dcf2212ad88fd6b2a553ecb62dfbeeedc68: Status 404 returned error can't find the container with id 9b3ae207a4d9d88cc30244746f0a4dcf2212ad88fd6b2a553ecb62dfbeeedc68 Oct 30 01:53:18.181880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.181840 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:18.181984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.181909 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:18.181984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.181939 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:18.336543 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.336503 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:18.336709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.336569 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:18.374767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.374729 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" event={"ID":"ac7ecb81-e269-40d7-a976-b548e71d50b9","Type":"ContainerStarted","Data":"2fbcf8891ad110dbdcec0902b344c1b07de211cd2d87dc798a3ccb7f61590373"} Oct 30 01:53:18.374930 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.374779 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" event={"ID":"ac7ecb81-e269-40d7-a976-b548e71d50b9","Type":"ContainerStarted","Data":"9b3ae207a4d9d88cc30244746f0a4dcf2212ad88fd6b2a553ecb62dfbeeedc68"} Oct 30 01:53:18.418610 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.418549 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podStartSLOduration=1.418532004 podStartE2EDuration="1.418532004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:18.416332479 +0000 UTC m=+1426.808140110" watchObservedRunningTime="2026-10-30 01:53:18.418532004 +0000 UTC m=+1426.810339708" Oct 30 01:53:18.807178 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:18.806229 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:19.336219 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:19.336180 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:53:19.336402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:19.336246 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:53:19.808178 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:19.808102 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:53:19.808666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:19.808203 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 01:53:20.805911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:20.805878 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:53:20.806182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:20.806145 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 01:53:21.336349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:21.336291 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:21.336759 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:21.336368 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:21.806651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:21.806605 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:53:21.806873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:21.806663 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 01:53:22.221850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:22.221770 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:22.222830 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:22.222807 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:22.225498 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:22.225475 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 01:53:22.806789 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:22.806753 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:53:22.807224 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:22.806807 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 01:53:23.805945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:23.805909 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:53:23.806195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:23.805970 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 01:53:24.806670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:24.806634 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:53:24.807067 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:24.806688 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 01:53:25.825113 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:25.825078 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:25.825794 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:25.825773 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:25.826728 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:25.826705 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 01:53:30.179147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:30.178199 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 01:53:30.547189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:30.529212 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.699147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:30.687267 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.088146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.079312 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.194145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.182218 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.194145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.183077 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:53:31.537323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.537272 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.542490 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.538927 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.558146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.546537 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:53:31.736363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:31.736306 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:53:32.079622 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:32.079556 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.213148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:32.212243 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.306159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:32.290162 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:53:32.306159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:32.291530 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:53:32.551148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:32.550375 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.755169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:32.744232 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:33.303142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.301484 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 01:53:33.318175 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.315599 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:33.408254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.402460 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:33.408254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.402531 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:33.431950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.426492 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:53:33.534139 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.533284 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:53:33.541204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.534812 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:53:33.568191 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:33.567623 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:53:40.970823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:40.970783 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s"] Oct 30 01:53:40.976222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:40.976199 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.005654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.005623 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s"] Oct 30 01:53:41.098915 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.098871 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.107302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.107260 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.117542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.117271 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.117542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.117339 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.117825 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.117568 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.118272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.117665 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.118429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.118307 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mqp\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.118429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.118351 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.118532 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.118440 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.219863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.219826 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.219863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.219875 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.219907 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.219953 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.219996 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220160 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mqp\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220414 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220236 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220414 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220274 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220414 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220373 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220414 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220402 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220861 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.220931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220899 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.221048 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.220991 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.221048 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.221013 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.222573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.222550 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.222643 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.222590 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.242535 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.242500 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.242658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.242600 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mqp\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.295227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.295191 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:41.638657 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.638628 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s"] Oct 30 01:53:41.641424 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:53:41.641398 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b8cee0_776d_4ae7_a646_d3060e964bb1.slice/crio-bc832f37f0d88e5312c4724d26dbe623de1a83a6eb1adac572e8a5258102d6fa WatchSource:0}: Error finding container bc832f37f0d88e5312c4724d26dbe623de1a83a6eb1adac572e8a5258102d6fa: Status 404 returned error can't find the container with id bc832f37f0d88e5312c4724d26dbe623de1a83a6eb1adac572e8a5258102d6fa Oct 30 01:53:41.643873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.643829 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:41.643962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.643911 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:41.643962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.643947 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:41.923417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.923295 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" event={"ID":"16b8cee0-776d-4ae7-a646-d3060e964bb1","Type":"ContainerStarted","Data":"602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf"} Oct 30 01:53:41.923726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:41.923706 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" event={"ID":"16b8cee0-776d-4ae7-a646-d3060e964bb1","Type":"ContainerStarted","Data":"bc832f37f0d88e5312c4724d26dbe623de1a83a6eb1adac572e8a5258102d6fa"} Oct 30 01:53:42.300145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:42.299520 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:42.312165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:42.306105 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:42.312165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:42.306187 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:43.296615 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:43.296569 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:43.297000 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:43.296964 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:44.245178 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.237288 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:44.245178 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.237358 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.297087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.296584 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:44.297087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.296649 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:44.594376 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.594328 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.667292 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.667253 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:44.667611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.667561 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.777146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:44.773100 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.285259 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:45.257276 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.304164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:45.296583 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:45.304164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:45.296652 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:45.612415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:45.612315 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.668105 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:45.667594 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:45.668105 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:45.667667 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.258798 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.257485 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.261503 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.259483 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:53:46.304172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.303677 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podStartSLOduration=6.303656478 podStartE2EDuration="6.303656478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:41.974180759 +0000 UTC m=+1450.365988412" watchObservedRunningTime="2026-10-30 01:53:46.303656478 +0000 UTC m=+1454.695464115" Oct 30 01:53:46.306292 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.306259 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:46.306553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.306527 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:46.633362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.633262 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.707753 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:46.707724 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:53:46.769706 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:53:46.769666 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:53:47.296554 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:47.296515 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:47.296732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:47.296574 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:48.295933 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:48.295898 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:48.296429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:48.295953 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:49.297091 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:49.297052 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:49.298606 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:49.298565 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:50.296237 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:50.296196 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:50.296408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:50.296259 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:51.296276 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:51.296234 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:53:51.296776 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:51.296296 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 01:53:52.300696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:52.300665 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:52.301201 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:52.301083 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:52.301951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:52.301927 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:53:58.193682 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.193641 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2"] Oct 30 01:53:58.198419 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.198394 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.213079 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.213050 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2933ff00-openshift-default-dockercfg-h4nbj\"" Oct 30 01:53:58.234695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.234663 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2"] Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.320184 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.320400 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.320708 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.320831 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.320859 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.320920 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrc6\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.321030 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.321078 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.322157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.321101 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.422949 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423033 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrc6\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423151 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423197 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423220 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423294 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423391 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423702 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423473 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.423702 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.423546 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.426143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.424058 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.449192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.448945 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.454841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.449570 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.454841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.450332 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.454841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.450842 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.462644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.462617 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.466815 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.466725 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.482697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.476017 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrc6\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.496147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.495177 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.517817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.516388 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:53:58.882808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.882782 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2"] Oct 30 01:53:58.885205 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:53:58.885175 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73873853_6186_40f3_bb4d_36aa727df94a.slice/crio-c66be585900a79e3786b156d08232fd47d88570cf0276f09154051d83cfa1142 WatchSource:0}: Error finding container c66be585900a79e3786b156d08232fd47d88570cf0276f09154051d83cfa1142: Status 404 returned error can't find the container with id c66be585900a79e3786b156d08232fd47d88570cf0276f09154051d83cfa1142 Oct 30 01:53:58.887664 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.887630 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:58.887772 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.887709 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:58.887772 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:58.887752 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:53:59.389656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.389614 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns"] Oct 30 01:53:59.394202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.394180 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.407873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.407843 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2198e4-openshift-default-dockercfg-zhwrb\"" Oct 30 01:53:59.421652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.421621 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns"] Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560176 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560244 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560286 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560402 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560456 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghsnx\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560487 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560616 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560646 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.564424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.560686 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.622146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.619170 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" event={"ID":"73873853-6186-40f3-bb4d-36aa727df94a","Type":"ContainerStarted","Data":"6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0"} Oct 30 01:53:59.622146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.619242 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" event={"ID":"73873853-6186-40f3-bb4d-36aa727df94a","Type":"ContainerStarted","Data":"c66be585900a79e3786b156d08232fd47d88570cf0276f09154051d83cfa1142"} Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.664959 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665017 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665051 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665165 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665204 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665254 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665372 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665441 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghsnx\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.665473 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.666664 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.669206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.666946 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.678740 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.677925 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podStartSLOduration=1.677904967 podStartE2EDuration="1.677904967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:59.677452794 +0000 UTC m=+1468.069260428" watchObservedRunningTime="2026-10-30 01:53:59.677904967 +0000 UTC m=+1468.069712600" Oct 30 01:53:59.678740 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.678164 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.678740 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.678437 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.679358 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.679336 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.683160 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.681907 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.691460 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.690903 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.691460 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.691370 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.707342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.707176 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghsnx\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:53:59.722163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:53:59.720572 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:54:00.221712 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.221686 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns"] Oct 30 01:54:00.223777 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:54:00.223747 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc4b32b_8457_4756_ad2c_2e33d1ee93c5.slice/crio-5049e487d83486672bd300206f1ef5d761c8f76c62e0f2e0e34e3488e41466e4 WatchSource:0}: Error finding container 5049e487d83486672bd300206f1ef5d761c8f76c62e0f2e0e34e3488e41466e4: Status 404 returned error can't find the container with id 5049e487d83486672bd300206f1ef5d761c8f76c62e0f2e0e34e3488e41466e4 Oct 30 01:54:00.226596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.226557 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:00.226684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.226653 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:00.226719 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.226699 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:00.518794 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.518764 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:54:00.525505 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.525473 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:54:00.640592 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.640552 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" event={"ID":"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5","Type":"ContainerStarted","Data":"0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5"} Oct 30 01:54:00.640790 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.640639 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" event={"ID":"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5","Type":"ContainerStarted","Data":"5049e487d83486672bd300206f1ef5d761c8f76c62e0f2e0e34e3488e41466e4"} Oct 30 01:54:00.641102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.641069 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:54:00.644069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.644045 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:54:00.703250 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.700033 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podStartSLOduration=1.70001136 podStartE2EDuration="1.70001136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:00.683226827 +0000 UTC m=+1469.075034483" watchObservedRunningTime="2026-10-30 01:54:00.70001136 +0000 UTC m=+1469.091818994" Oct 30 01:54:00.726242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:00.722800 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:54:01.832146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:01.830584 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:54:02.461147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:02.460350 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.699144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:02.698841 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:54:02.708646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:02.708362 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:54:02.714370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:02.713209 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.756162 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:02.749327 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:54:03.243144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:03.242243 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:03.243144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:03.242351 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.464143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:03.462238 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.464143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:03.463331 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:54:03.584437 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:03.581689 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.288078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:04.287371 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:04.288078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:04.287438 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.541199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:04.506187 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:04.586170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:04.586110 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.587609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:04.587583 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:54:05.242143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.214829 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 01:54:05.332286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.332243 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:05.332912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.332879 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.333559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.333533 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:05.333748 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.333725 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.390160 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.389086 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.593147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.568529 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.706157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.644946 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.857926 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:05.857827 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.284283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.284234 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.442142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.439902 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:54:06.616283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.594700 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:06.745949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.745905 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 01:54:06.819742 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.819711 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:54:06.906386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.906218 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:06.906386 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.906283 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.982931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:06.982875 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.028185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.024929 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4"] Oct 30 01:54:07.140391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.140352 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4"] Oct 30 01:54:07.150600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.141412 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.214210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.200488 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2933ff00\"" Oct 30 01:54:07.378159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.273301 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2933ff00-dockercfg-tgxf9\"" Oct 30 01:54:07.403998 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.403961 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.404379 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.404352 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tsv\" (UniqueName: \"kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.506610 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.506518 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tsv\" (UniqueName: \"kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.507436 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.507404 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.508547 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.508523 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.582189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.582060 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tsv\" (UniqueName: \"kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.668272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.668224 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.695394 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.695356 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:07.793161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:07.776704 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:08.266647 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:08.252496 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:54:08.564076 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:08.563978 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:08.578153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:08.577849 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:08.631705 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:08.630570 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:54:09.072323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:09.049576 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4"] Oct 30 01:54:09.558328 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:09.558290 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" event={"ID":"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5","Type":"ContainerStarted","Data":"a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9"} Oct 30 01:54:09.558523 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:09.558374 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" event={"ID":"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5","Type":"ContainerStarted","Data":"43d7f4a2065f51ed97cc1311c31e442615d28e3460a11be3ded8821b70d68721"} Oct 30 01:54:09.559276 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:09.559253 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:16.697511 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:54:16.697475 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:54:20.209507 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:20.207555 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:20.209507 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:20.207645 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.621272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:20.621209 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.743140 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:20.739380 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.767183 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:20.767141 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:54:20.824292 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:20.824228 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.478148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.469400 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" podStartSLOduration=15.469376639 podStartE2EDuration="15.469376639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:09.608342145 +0000 UTC m=+1478.000149778" watchObservedRunningTime="2026-10-30 01:54:21.469376639 +0000 UTC m=+1489.861184273" Oct 30 01:54:21.648174 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.648114 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.665034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.650489 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 01:54:21.665034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.652000 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Oct 30 01:54:21.665034 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.652099 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:54:21.706158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.704411 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:54:21.845176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.789303 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.919439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.849088 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:22.215200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:21.855157 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:54:22.652691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:22.651332 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:23.113360 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:54:23.098571 6275 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.334s" Oct 30 01:54:23.113360 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.098627 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5"] Oct 30 01:54:23.283156 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.283106 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:54:23.283406 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.283387 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5"] Oct 30 01:54:23.364746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.357966 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 01:54:23.394246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.394218 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:54:23.399283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.399258 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:54:23.414442 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.414413 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 01:54:23.414770 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.414750 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 01:54:23.414927 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.394536 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.573148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.534148 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.653650 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.606593 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 01:54:23.654393 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.616528 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2198e4-serving-cert\"" Oct 30 01:54:23.654579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.630254 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:23.654745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.653492 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.654907 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.654895 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 01:54:23.655288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.655253 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:23.659925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.659901 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478c7\" (UniqueName: \"kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.668009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.667734 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.821058 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.782831 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.821058 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.782962 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.821058 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.783018 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.821058 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.783235 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478c7\" (UniqueName: \"kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.866146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.844268 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.965650 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.917675 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:23.965650 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:23.932951 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:24.044523 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:24.027469 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-478c7\" (UniqueName: \"kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7\") pod \"maas-api-e2e-api-a-2198e4-75d8794795-b6pc5\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:24.044523 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:24.031346 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:24.550346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:24.550016 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5"] Oct 30 01:54:24.552815 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:54:24.552765 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4e78d9_0f08_41ec_bd29_a9f172abb960.slice/crio-1d0f13ad3a7406742bd0ee236a8ee23946546990244a24846c544c7b254483e5 WatchSource:0}: Error finding container 1d0f13ad3a7406742bd0ee236a8ee23946546990244a24846c544c7b254483e5: Status 404 returned error can't find the container with id 1d0f13ad3a7406742bd0ee236a8ee23946546990244a24846c544c7b254483e5 Oct 30 01:54:24.555825 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:24.555485 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:54:25.502870 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:25.502834 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" event={"ID":"5f4e78d9-0f08-41ec-bd29-a9f172abb960","Type":"ContainerStarted","Data":"47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491"} Oct 30 01:54:25.503395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:25.502892 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" event={"ID":"5f4e78d9-0f08-41ec-bd29-a9f172abb960","Type":"ContainerStarted","Data":"1d0f13ad3a7406742bd0ee236a8ee23946546990244a24846c544c7b254483e5"} Oct 30 01:54:25.503395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:25.503033 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:25.548313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:25.548257 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" podStartSLOduration=4.140077557 podStartE2EDuration="4.548242688s" totalImagesPullingTime="408.165131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:25.545773219 +0000 UTC m=+1493.937580852" watchObservedRunningTime="2026-10-30 01:54:25.548242688 +0000 UTC m=+1493.940050321" Oct 30 01:54:26.354288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.354253 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522"] Oct 30 01:54:26.359049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.359025 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.391905 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.391864 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522"] Oct 30 01:54:26.412784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.412744 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.412973 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.412800 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.412973 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.412891 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.413082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.412973 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.413082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.413031 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.413082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.413071 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.413236 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.413223 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2pt\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.413431 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.413410 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.413518 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.413469 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515106 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515191 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515236 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515267 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515292 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515321 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515366 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515417 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515484 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2pt\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.515641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515621 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.516087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.515972 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.516183 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.516156 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.516255 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.516226 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.516545 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.516521 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.517783 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.517763 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.518646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.518627 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.538345 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.538279 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.539245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.539219 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2pt\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt\") pod \"maas-default-gateway-openshift-default-75d7c785f-nw522\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:26.674794 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:26.674715 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:27.032969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.032850 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522"] Oct 30 01:54:27.047140 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.045988 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:27.047140 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.046108 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:27.047140 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.046185 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:27.533626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.533564 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" event={"ID":"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4","Type":"ContainerStarted","Data":"64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410"} Oct 30 01:54:27.534075 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.533711 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" event={"ID":"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4","Type":"ContainerStarted","Data":"50144469352ce75793dfdd57e3835733d96e3b2b68a4ec15b0be635153258e29"} Oct 30 01:54:27.605072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.605006 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podStartSLOduration=1.604987943 podStartE2EDuration="1.604987943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:27.59400325 +0000 UTC m=+1495.985810884" watchObservedRunningTime="2026-10-30 01:54:27.604987943 +0000 UTC m=+1495.996795577" Oct 30 01:54:27.674872 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:27.674837 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:28.676342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:28.676305 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:28.676753 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:28.676373 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:29.675150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:29.675103 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:29.675342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:29.675172 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:30.675387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:30.675352 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:30.675801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:30.675417 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:31.516012 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:31.515926 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:54:31.676038 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:31.676007 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:31.676457 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:31.676054 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:31.967658 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:31.967576 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:54:31.968319 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:31.968296 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:54:32.675911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:32.675878 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:32.676106 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:32.675930 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:33.675029 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:33.674993 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:33.675258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:33.675058 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:34.676223 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:34.676184 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nw522 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:54:34.676734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:34.676241 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:54:36.118657 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:36.118624 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:36.120011 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:36.119987 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:36.120764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:36.120747 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 01:54:41.172687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.172649 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr"] Oct 30 01:54:41.178340 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.178317 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.209847 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.209813 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr"] Oct 30 01:54:41.258033 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.257995 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258033 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258038 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258177 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258332 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258307 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdkcl\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258385 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258369 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258425 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258399 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258511 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258492 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258563 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258551 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.258605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.258587 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364142 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364246 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdkcl\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364297 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364319 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364385 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364465 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364488 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364640 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.364676 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.365562 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.371182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.365896 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.388149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.382325 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.388149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.387812 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.391322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.391278 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.416045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.416011 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.416286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.416204 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.416938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.416855 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdkcl\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.418765 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.418740 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.492450 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.492362 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:41.822823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.822798 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr"] Oct 30 01:54:41.826341 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:54:41.826311 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19718249_1de6_4aad_a8a7_15c3cb5c41b8.slice/crio-e695df7d8ad597e36b6371222bcaf7032774ce16f6dca22c28be1600c9013845 WatchSource:0}: Error finding container e695df7d8ad597e36b6371222bcaf7032774ce16f6dca22c28be1600c9013845: Status 404 returned error can't find the container with id e695df7d8ad597e36b6371222bcaf7032774ce16f6dca22c28be1600c9013845 Oct 30 01:54:41.828914 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.828884 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:41.828984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.828949 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:41.828984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:41.828979 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:42.674599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:42.672094 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" event={"ID":"19718249-1de6-4aad-a8a7-15c3cb5c41b8","Type":"ContainerStarted","Data":"e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25"} Oct 30 01:54:42.674599 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:42.672197 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" event={"ID":"19718249-1de6-4aad-a8a7-15c3cb5c41b8","Type":"ContainerStarted","Data":"e695df7d8ad597e36b6371222bcaf7032774ce16f6dca22c28be1600c9013845"} Oct 30 01:54:42.734218 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:42.734160 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podStartSLOduration=1.734142383 podStartE2EDuration="1.734142383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:42.731810317 +0000 UTC m=+1511.123617956" watchObservedRunningTime="2026-10-30 01:54:42.734142383 +0000 UTC m=+1511.125950017" Oct 30 01:54:43.493094 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:43.493055 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:43.494564 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:43.494536 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:43.494695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:43.494597 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:44.493403 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.493358 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:44.493857 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.493437 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:44.497345 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.497318 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx"] Oct 30 01:54:44.501670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.501645 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.536907 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.536863 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx"] Oct 30 01:54:44.628410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.628350 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.628758 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.628732 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7mn\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.628937 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.628922 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.629137 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.629101 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.629465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.629427 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.629654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.629637 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.629806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.629792 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.629938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.629922 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.630083 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.630070 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.731704 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.731669 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.731913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.731797 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.731913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.731850 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732023 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.731999 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732085 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732035 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732156 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732113 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732156 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732132 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732210 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732314 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732241 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732314 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732283 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7mn\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732331 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732570 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732548 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732624 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732566 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.732720 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.732700 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.734391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.734371 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.734776 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.734759 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.769331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.766651 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.773784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.773751 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7mn\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:44.816779 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:44.816740 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:45.164262 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.151067 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6"] Oct 30 01:54:45.323349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.301945 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6"] Oct 30 01:54:45.323349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.302066 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.353161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.326274 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c2993e-openshift-default-dockercfg-2skkn\"" Oct 30 01:54:45.457884 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.355481 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.517017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.379445 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx"] Oct 30 01:54:45.517017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.390576 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:45.517017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.502440 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:45.517017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.502507 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:45.519459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.518035 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:45.519459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.518096 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:45.519459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519263 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.519459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519321 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.519759 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519481 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.519759 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519560 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.519759 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519615 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.519759 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519752 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.520015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519785 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtdf\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.520015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.519846 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621403 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621472 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621576 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621674 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621657 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621727 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621710 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621737 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621793 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621770 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621836 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621819 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtdf\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.621988 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.621972 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.622040 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.622027 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.622183 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.622160 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.622346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.622332 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.622932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.622847 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.623037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.622997 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.625078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.625045 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.625323 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.625300 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.664785 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.664719 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.806328 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.806292 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtdf\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf\") pod \"e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.836436 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.836400 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:45.893189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.882962 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" event={"ID":"a8881725-7a6b-41e2-905b-bfc750854b42","Type":"ContainerStarted","Data":"f1e8ac61e5f9ad1d91d1d29ec09fe59b2db818af09a594d7845f9d8baf6ab166"} Oct 30 01:54:45.916193 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:45.914441 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" event={"ID":"a8881725-7a6b-41e2-905b-bfc750854b42","Type":"ContainerStarted","Data":"e4ee7ed4f3b8eb046f9c24321edfd5eea3d543f89c30f508beaa44fb1e120228"} Oct 30 01:54:46.371684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.371626 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podStartSLOduration=2.371532849 podStartE2EDuration="2.371532849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:45.974522649 +0000 UTC m=+1514.366330285" watchObservedRunningTime="2026-10-30 01:54:46.371532849 +0000 UTC m=+1514.763340482" Oct 30 01:54:46.375353 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:54:46.375323 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03107880_070a_4f45_b875_b1cd7468fe2a.slice/crio-c161d07121644a040052efac86fd042954eb188070e63c2a5d16eff21666a147 WatchSource:0}: Error finding container c161d07121644a040052efac86fd042954eb188070e63c2a5d16eff21666a147: Status 404 returned error can't find the container with id c161d07121644a040052efac86fd042954eb188070e63c2a5d16eff21666a147 Oct 30 01:54:46.376049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.376024 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6"] Oct 30 01:54:46.378844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.378807 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:46.379002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.378896 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:46.379002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.378941 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:54:46.493640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.493523 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:46.493640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.493582 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:46.824228 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.824190 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:46.824955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.824924 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:46.825067 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.824981 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:46.827491 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:54:46.827462 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:54:46.936147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.921039 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" event={"ID":"03107880-070a-4f45-b875-b1cd7468fe2a","Type":"ContainerStarted","Data":"7a45614bf5d39401946a19d64be45dde6aebf4651aad8ea1085b0caf27f7aed2"} Oct 30 01:54:46.936147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.921101 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" event={"ID":"03107880-070a-4f45-b875-b1cd7468fe2a","Type":"ContainerStarted","Data":"c161d07121644a040052efac86fd042954eb188070e63c2a5d16eff21666a147"} Oct 30 01:54:46.983717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:46.983589 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podStartSLOduration=1.983566085 podStartE2EDuration="1.983566085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:46.968745451 +0000 UTC m=+1515.360553084" watchObservedRunningTime="2026-10-30 01:54:46.983566085 +0000 UTC m=+1515.375373719" Oct 30 01:54:47.501149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.497948 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:47.501149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.498013 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:47.832147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.824524 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:47.832147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.824590 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:47.886495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.886456 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:47.886990 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.886967 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:47.958071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.958039 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:47.961596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:47.961567 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:54:48.525157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.521702 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:48.525157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.521768 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:48.636036 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.635988 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:48.660222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.660164 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:48.735147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.731185 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:48.833619 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.833524 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:48.834211 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:48.834176 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:49.496155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:49.495573 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:49.496155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:49.495652 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:54:49.913420 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:49.912772 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:49.913420 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:49.912842 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:50.833152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:50.821661 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:51.049574 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.049524 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:51.499669 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.499630 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:51.500092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.500057 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:51.637545 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.637500 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:51.672169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.670702 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:51.672777 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.672749 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:51.686665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.686632 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:54:51.977142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.976411 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:51.977142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:51.976491 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:52.236956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.236874 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl"] Oct 30 01:54:52.345465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.345430 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl"] Oct 30 01:54:52.345854 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.345837 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.391565 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.391535 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c2993e\"" Oct 30 01:54:52.489487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.478814 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzkj\" (UniqueName: \"kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj\") pod \"payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.489487 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.478877 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.499104 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.499063 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd"] Oct 30 01:54:52.570580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.570550 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:52.608151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.604542 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzkj\" (UniqueName: \"kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj\") pod \"payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.612152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.608424 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.612429 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.612397 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgpt\" (UniqueName: \"kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt\") pod \"payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:52.626267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.612594 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:52.626845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.617468 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.627004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.624825 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c2993e-dockercfg-vld7z\"" Oct 30 01:54:52.668549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.668457 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd"] Oct 30 01:54:52.754544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.754455 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:52.759146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.756046 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgpt\" (UniqueName: \"kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt\") pod \"payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:52.784326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.755620 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:52.784576 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.775822 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzkj\" (UniqueName: \"kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj\") pod \"payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.795002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.794971 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:52.818070 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.818038 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:52.818331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.818302 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:52.820199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:52.819977 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgpt\" (UniqueName: \"kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt\") pod \"payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:53.031998 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:53.031762 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:53.830485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:53.828204 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:53.831102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:53.831056 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:54:53.890149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:53.863987 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl"] Oct 30 01:54:54.007407 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.006961 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd"] Oct 30 01:54:54.012410 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:54:54.012379 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100651e3_344b_4845_8f08_9f704d047361.slice/crio-e0093540e0c425a897f18126a0cfeb5d81865dd547bf96f40abef925cecc571f WatchSource:0}: Error finding container e0093540e0c425a897f18126a0cfeb5d81865dd547bf96f40abef925cecc571f: Status 404 returned error can't find the container with id e0093540e0c425a897f18126a0cfeb5d81865dd547bf96f40abef925cecc571f Oct 30 01:54:54.546844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.546765 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" event={"ID":"100651e3-344b-4845-8f08-9f704d047361","Type":"ContainerStarted","Data":"832c6b86774cd0642db6e1231d6c2d322dc6a87bc01bb632b223150922f9ceec"} Oct 30 01:54:54.546844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.546831 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:54:54.547080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.546853 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" event={"ID":"100651e3-344b-4845-8f08-9f704d047361","Type":"ContainerStarted","Data":"e0093540e0c425a897f18126a0cfeb5d81865dd547bf96f40abef925cecc571f"} Oct 30 01:54:54.548731 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.548698 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" event={"ID":"c32b132a-05bd-48da-8cfd-d9047ca96316","Type":"ContainerStarted","Data":"7f8e2380a51503bdcd5f448ce95e875fb24e9ed1294a524dacffc659f5e9611d"} Oct 30 01:54:54.548873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.548754 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" event={"ID":"c32b132a-05bd-48da-8cfd-d9047ca96316","Type":"ContainerStarted","Data":"5c9c9ee64d97d536e68f445ea591d3514b3d312ad48158c9a101515489d18c97"} Oct 30 01:54:54.548873 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.548798 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:54:54.586835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.586179 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" podStartSLOduration=2.586161327 podStartE2EDuration="2.586161327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:54.584263056 +0000 UTC m=+1522.976070715" watchObservedRunningTime="2026-10-30 01:54:54.586161327 +0000 UTC m=+1522.977968958" Oct 30 01:54:54.627361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:54.627293 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" podStartSLOduration=2.627274061 podStartE2EDuration="2.627274061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:54.621572163 +0000 UTC m=+1523.013379797" watchObservedRunningTime="2026-10-30 01:54:54.627274061 +0000 UTC m=+1523.019081695" Oct 30 01:54:55.410746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:55.410709 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:55.411227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:55.411166 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:55.412631 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:55.412606 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 01:54:59.553783 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.553748 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl"] Oct 30 01:54:59.578199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.578036 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.596784 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.596755 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl"] Oct 30 01:54:59.642500 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642397 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.642500 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642497 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwhc\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.642812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642529 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.642812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642562 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.642812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642618 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.642812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642794 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.643030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642843 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.643030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642898 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.643030 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.642949 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744017 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744255 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744061 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744255 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744115 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744334 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744423 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744408 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744479 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744672 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744842 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744854 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744961 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744934 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.744961 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744940 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.745042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.744965 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.745095 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.745062 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwhc\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.745196 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.745178 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.747164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.747135 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.747403 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.747383 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.778741 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.778708 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.784170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.784109 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwhc\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:54:59.928405 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:54:59.928379 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:00.285961 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.285935 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl"] Oct 30 01:55:00.288912 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:55:00.288882 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c68408_9cde_40e8_af62_8d0814b8101b.slice/crio-c02cdd9b0c11afc1fb37a64a7731e133aa44df20cb376f3f4c3a7e6faab0f2b7 WatchSource:0}: Error finding container c02cdd9b0c11afc1fb37a64a7731e133aa44df20cb376f3f4c3a7e6faab0f2b7: Status 404 returned error can't find the container with id c02cdd9b0c11afc1fb37a64a7731e133aa44df20cb376f3f4c3a7e6faab0f2b7 Oct 30 01:55:00.291584 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.291549 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.291666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.291620 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.291666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.291650 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.614597 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.614509 6275 generic.go:356] "Generic (PLEG): container finished" podID="73873853-6186-40f3-bb4d-36aa727df94a" containerID="6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0" exitCode=0 Oct 30 01:55:00.615065 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.614587 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" event={"ID":"73873853-6186-40f3-bb4d-36aa727df94a","Type":"ContainerDied","Data":"6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0"} Oct 30 01:55:00.615225 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.615200 6275 scope.go:122] "RemoveContainer" containerID="6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0" Oct 30 01:55:00.616491 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.616469 6275 generic.go:356] "Generic (PLEG): container finished" podID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerID="0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5" exitCode=0 Oct 30 01:55:00.616607 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.616557 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" event={"ID":"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5","Type":"ContainerDied","Data":"0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5"} Oct 30 01:55:00.616967 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.616935 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.617074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.616969 6275 scope.go:122] "RemoveContainer" containerID="0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5" Oct 30 01:55:00.617074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.617022 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.617074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.617072 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.618863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.618833 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.618939 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.618918 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.618982 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.618962 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:00.619378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.619352 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" event={"ID":"67c68408-9cde-40e8-af62-8d0814b8101b","Type":"ContainerStarted","Data":"ad4d26e729fc1510fb4503563c427d0cfc7b24db4b0bcb27f90cf7a10c66e3fe"} Oct 30 01:55:00.619481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.619393 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" event={"ID":"67c68408-9cde-40e8-af62-8d0814b8101b","Type":"ContainerStarted","Data":"c02cdd9b0c11afc1fb37a64a7731e133aa44df20cb376f3f4c3a7e6faab0f2b7"} Oct 30 01:55:00.641659 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.641635 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:55:00.775365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.767992 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podStartSLOduration=1.767970021 podStartE2EDuration="1.767970021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:00.756649621 +0000 UTC m=+1529.148457256" watchObservedRunningTime="2026-10-30 01:55:00.767970021 +0000 UTC m=+1529.159777655" Oct 30 01:55:00.928956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:00.928842 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:01.636656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.634995 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" event={"ID":"73873853-6186-40f3-bb4d-36aa727df94a","Type":"ContainerStarted","Data":"225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e"} Oct 30 01:55:01.637495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.637473 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" event={"ID":"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5","Type":"ContainerStarted","Data":"d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589"} Oct 30 01:55:01.722573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.722531 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:55:01.729526 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.729488 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:55:01.729723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.729550 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:55:01.930594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.930500 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:55:01.930594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:01.930573 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:55:02.518797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.518716 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:55:02.520338 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.520305 6275 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.520509 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.520371 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:55:02.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.696217 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:55:02.725641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.722448 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.725641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.722506 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:55:02.929094 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.928996 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.929094 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:02.929073 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:55:03.517660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:03.517624 6275 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.517889 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:03.517709 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:55:03.726166 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:03.725559 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.726166 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:03.725615 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:55:03.929875 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:03.929783 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.929875 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:03.929850 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:55:04.522151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:04.521506 6275 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.522151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:04.521569 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:55:04.725970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:04.725512 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.725970 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:04.725568 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:55:04.928848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:04.928761 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.928848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:04.928813 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:55:05.517692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.517635 6275 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:55:05.517875 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.517712 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:55:05.595147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.594282 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:55:05.616148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.606271 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:55:05.726149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.724717 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:55:05.726149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.724781 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:55:05.934177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.934050 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:55:05.934177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:05.934112 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:55:06.425842 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.412217 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z"] Oct 30 01:55:06.487270 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.463370 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z"] Oct 30 01:55:06.487568 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.482323 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.512538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.512504 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c2993e-serving-cert\"" Oct 30 01:55:06.520626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.520592 6275 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:55:06.520889 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.520841 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:55:06.537008 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.536978 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.537186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.537156 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.537264 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.537189 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.537332 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.537315 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f4k\" (UniqueName: \"kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.747547 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.642742 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.765860 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.765749 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.766064 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.765909 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f4k\" (UniqueName: \"kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.766221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.766149 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.767080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.767002 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.767251 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.738675 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:55:06.767378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.767358 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:55:06.768305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.747090 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.859098 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.859052 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f4k\" (UniqueName: \"kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.913920 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.909751 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls\") pod \"maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:06.928867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.928832 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:55:06.929159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:06.929111 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:55:07.136926 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.136899 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:07.512255 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.512217 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z"] Oct 30 01:55:07.524711 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.524689 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:55:07.525002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.524982 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:55:07.525682 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.525663 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:55:07.855879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.855790 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:55:07.857540 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.857521 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:55:07.933935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.933910 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:07.934213 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.934192 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:07.934932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.934913 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:07.943508 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:07.943483 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" event={"ID":"db4e0f1b-f85c-412f-9e8d-c26b38557344","Type":"ContainerStarted","Data":"e8574bbcb1c01abdff38a1ee8db79e51b7bd3236bb7c8afcd458ad05050aeb45"} Oct 30 01:55:08.959363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:08.959323 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" event={"ID":"db4e0f1b-f85c-412f-9e8d-c26b38557344","Type":"ContainerStarted","Data":"b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc"} Oct 30 01:55:08.959999 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:08.959970 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:09.019818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:09.019758 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" podStartSLOduration=2.612657178 podStartE2EDuration="3.019744999s" totalImagesPullingTime="407.087821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:09.018552384 +0000 UTC m=+1537.410360017" watchObservedRunningTime="2026-10-30 01:55:09.019744999 +0000 UTC m=+1537.411552631" Oct 30 01:55:14.971040 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:14.970963 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:55:16.697250 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:16.697213 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:55:38.068844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.068803 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld"] Oct 30 01:55:38.072450 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.072422 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.096068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.096032 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld"] Oct 30 01:55:38.102481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.102452 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-922b2c\"" Oct 30 01:55:38.102627 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.102531 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-922b2c-dockercfg-bj2bx\"" Oct 30 01:55:38.209143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.182591 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.209143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.182945 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxp6w\" (UniqueName: \"kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w\") pod \"payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.366975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.333475 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8"] Oct 30 01:55:38.366975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.340490 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxp6w\" (UniqueName: \"kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w\") pod \"payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.366975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.340687 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.366975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.341933 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.403246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.402287 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8"] Oct 30 01:55:38.403246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.402405 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.439212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.438834 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxp6w\" (UniqueName: \"kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w\") pod \"payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.440752 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.440531 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:38.498520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.493737 6275 generic.go:356] "Generic (PLEG): container finished" podID="03107880-070a-4f45-b875-b1cd7468fe2a" containerID="7a45614bf5d39401946a19d64be45dde6aebf4651aad8ea1085b0caf27f7aed2" exitCode=0 Oct 30 01:55:38.498520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.494037 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" event={"ID":"03107880-070a-4f45-b875-b1cd7468fe2a","Type":"ContainerDied","Data":"7a45614bf5d39401946a19d64be45dde6aebf4651aad8ea1085b0caf27f7aed2"} Oct 30 01:55:38.498520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.494704 6275 scope.go:122] "RemoveContainer" containerID="7a45614bf5d39401946a19d64be45dde6aebf4651aad8ea1085b0caf27f7aed2" Oct 30 01:55:38.498520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.496808 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:38.498520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.496875 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:38.498520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.496909 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:38.555965 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.555923 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkzj\" (UniqueName: \"kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj\") pod \"payload-processing-e2e-api-a-922b2c-66bf787d66-msht8\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.556072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.556000 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922b2c-66bf787d66-msht8\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.667210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.665254 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkzj\" (UniqueName: \"kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj\") pod \"payload-processing-e2e-api-a-922b2c-66bf787d66-msht8\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.667210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.665325 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922b2c-66bf787d66-msht8\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.667210 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.666542 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922b2c-66bf787d66-msht8\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.689756 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.689717 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkzj\" (UniqueName: \"kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj\") pod \"payload-processing-e2e-api-a-922b2c-66bf787d66-msht8\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.792565 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.792531 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:38.816776 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:38.816744 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld"] Oct 30 01:55:38.819114 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:55:38.819079 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f22161c_ec4c_46bd_9503_fe3eabb96b6f.slice/crio-fb614198a8c30ac86bb8a3b610c1c0711d779eff61f2cfd277c7a197ba3fa4f0 WatchSource:0}: Error finding container fb614198a8c30ac86bb8a3b610c1c0711d779eff61f2cfd277c7a197ba3fa4f0: Status 404 returned error can't find the container with id fb614198a8c30ac86bb8a3b610c1c0711d779eff61f2cfd277c7a197ba3fa4f0 Oct 30 01:55:39.154481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.154439 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8"] Oct 30 01:55:39.502037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.501949 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" event={"ID":"5f22161c-ec4c-46bd-9503-fe3eabb96b6f","Type":"ContainerStarted","Data":"387ac8259702d77363e6db38c2452928b6bc746566956ecb048c47d4eadce0bb"} Oct 30 01:55:39.502037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.501999 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" event={"ID":"5f22161c-ec4c-46bd-9503-fe3eabb96b6f","Type":"ContainerStarted","Data":"fb614198a8c30ac86bb8a3b610c1c0711d779eff61f2cfd277c7a197ba3fa4f0"} Oct 30 01:55:39.502329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.502146 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:39.503728 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.503703 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" event={"ID":"09db4327-de14-4608-8793-ac3a294aec70","Type":"ContainerStarted","Data":"ceb5ffedaff423be0a5c4c009697f7eeafe8fd6b19072c410a79d7ef1d7ca0e9"} Oct 30 01:55:39.503845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.503741 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" event={"ID":"09db4327-de14-4608-8793-ac3a294aec70","Type":"ContainerStarted","Data":"b1c2701c2fb01b96d7a9f9e70145ab4590239bc7da860381226985dad1b4fb4f"} Oct 30 01:55:39.503845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.503777 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:39.505717 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.505696 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" event={"ID":"03107880-070a-4f45-b875-b1cd7468fe2a","Type":"ContainerStarted","Data":"1b7f9b557d3c3e9aaf15d0d35ca77fb0c1ec93d866214bbe126dc4c96f55ca69"} Oct 30 01:55:39.565826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.565767 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" podStartSLOduration=1.565747121 podStartE2EDuration="1.565747121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:39.563573181 +0000 UTC m=+1567.955380815" watchObservedRunningTime="2026-10-30 01:55:39.565747121 +0000 UTC m=+1567.957554753" Oct 30 01:55:39.843697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.843667 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:55:39.844081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.844055 6275 patch_prober.go:29] interesting pod/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:39.844587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:39.844281 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 01:55:40.837362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:40.837325 6275 patch_prober.go:29] interesting pod/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:40.837741 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:40.837384 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 01:55:41.837894 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:41.837862 6275 patch_prober.go:29] interesting pod/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:41.838341 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:41.837916 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 01:55:42.837975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:42.837936 6275 patch_prober.go:29] interesting pod/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:42.838501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:42.837997 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 01:55:43.851154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:43.838230 6275 patch_prober.go:29] interesting pod/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:43.851154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:43.838289 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 01:55:44.496746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.496390 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" podStartSLOduration=6.496370727 podStartE2EDuration="6.496370727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:39.666765013 +0000 UTC m=+1568.058572678" watchObservedRunningTime="2026-10-30 01:55:44.496370727 +0000 UTC m=+1572.888178360" Oct 30 01:55:44.497516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.497491 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5"] Oct 30 01:55:44.497810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.497784 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" podUID="5f4e78d9-0f08-41ec-bd29-a9f172abb960" containerName="maas-api" containerID="cri-o://47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491" gracePeriod=30 Oct 30 01:55:44.592548 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.592512 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6"] Oct 30 01:55:44.596071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.596031 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.658563 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.658529 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6"] Oct 30 01:55:44.707460 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707424 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707525 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707571 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707637 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707685 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707766 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707801 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.707835 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707829 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.708039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.707979 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsg7g\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.815721 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.815677 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.816518 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.816495 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.817013 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.816986 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.817570 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.816895 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.817709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.816420 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.822267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.822236 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsg7g\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.822451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.822436 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.822652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.822637 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.822773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.822761 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.822958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.822945 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.823094 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.823081 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.835237 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.823657 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.835855 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.835825 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.856495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.838476 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.941146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.905085 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.986997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.986964 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.994334 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.993967 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:44.999063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:44.999033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsg7g\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:45.050293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.050258 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.196624 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca\") pod \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.196836 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7\" (UniqueName: \"kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7\") pod \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.196930 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls\") pod \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.196970 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs\") pod \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\" (UID: \"5f4e78d9-0f08-41ec-bd29-a9f172abb960\") " Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.222329 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.223044 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:55:45.226149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.223497 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca" pod "5f4e78d9-0f08-41ec-bd29-a9f172abb960" (UID: "5f4e78d9-0f08-41ec-bd29-a9f172abb960"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:45.240165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.235349 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:55:45.277204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.241429 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4e78d9-0f08-41ec-bd29-a9f172abb960-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:45.277204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.242642 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:45.300800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.299969 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls" pod "5f4e78d9-0f08-41ec-bd29-a9f172abb960" (UID: "5f4e78d9-0f08-41ec-bd29-a9f172abb960"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:45.300800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.300456 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs" pod "5f4e78d9-0f08-41ec-bd29-a9f172abb960" (UID: "5f4e78d9-0f08-41ec-bd29-a9f172abb960"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:45.344116 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.333838 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7" pod "5f4e78d9-0f08-41ec-bd29-a9f172abb960" (UID: "5f4e78d9-0f08-41ec-bd29-a9f172abb960"). InnerVolumeSpecName "kube-api-access-478c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:45.349190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.347184 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-478c7\" (UniqueName: \"kubernetes.io/projected/5f4e78d9-0f08-41ec-bd29-a9f172abb960-kube-api-access-478c7\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:45.349190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.347219 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:45.349190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.347237 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4e78d9-0f08-41ec-bd29-a9f172abb960-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:45.555515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.555439 6275 generic.go:356] "Generic (PLEG): container finished" podID="5f4e78d9-0f08-41ec-bd29-a9f172abb960" containerID="47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491" exitCode=0 Oct 30 01:55:45.555515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.555484 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" Oct 30 01:55:45.555694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.555518 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" event={"ID":"5f4e78d9-0f08-41ec-bd29-a9f172abb960","Type":"ContainerDied","Data":"47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491"} Oct 30 01:55:45.555694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.555575 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5" event={"ID":"5f4e78d9-0f08-41ec-bd29-a9f172abb960","Type":"ContainerDied","Data":"1d0f13ad3a7406742bd0ee236a8ee23946546990244a24846c544c7b254483e5"} Oct 30 01:55:45.555694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.555604 6275 scope.go:122] "RemoveContainer" containerID="47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491" Oct 30 01:55:45.566772 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.566753 6275 scope.go:122] "RemoveContainer" containerID="47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491" Oct 30 01:55:45.567049 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:45.567025 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491\": container with ID starting with 47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491 not found: ID does not exist" containerID="47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491" Oct 30 01:55:45.567116 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.567055 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491"} err="failed to get container status \"47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491\": rpc error: code = NotFound desc = could not find container \"47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491\": container with ID starting with 47c4f71bdc76faa8427f20ed343a02c6cc75e2d822737d5cc91ab82c42fa2491 not found: ID does not exist" Oct 30 01:55:45.610991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.610945 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5"] Oct 30 01:55:45.621881 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.621853 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2198e4-75d8794795-b6pc5"] Oct 30 01:55:45.700165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.692576 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6"] Oct 30 01:55:45.716313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.710025 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:45.716313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.710134 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:45.716313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.710182 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:55:45.765751 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:45.765718 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4e78d9-0f08-41ec-bd29-a9f172abb960" path="/var/lib/kubelet/pods/5f4e78d9-0f08-41ec-bd29-a9f172abb960/volumes" Oct 30 01:55:46.569354 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:46.569315 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" event={"ID":"d862bc10-3875-4427-9eee-1644c5e8e116","Type":"ContainerStarted","Data":"df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452"} Oct 30 01:55:46.569948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:46.569912 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" event={"ID":"d862bc10-3875-4427-9eee-1644c5e8e116","Type":"ContainerStarted","Data":"a3af5beb14ae6fe212cc4ea6188270176730dc24300dda9531023911fbd89209"} Oct 30 01:55:46.630410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:46.630359 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podStartSLOduration=2.630343248 podStartE2EDuration="2.630343248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:46.628238714 +0000 UTC m=+1575.020046348" watchObservedRunningTime="2026-10-30 01:55:46.630343248 +0000 UTC m=+1575.022150886" Oct 30 01:55:46.698572 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:46.698533 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:55:47.260549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:47.260515 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:47.264280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:47.264246 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:47.264414 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:47.264303 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:48.256147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:48.255563 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:48.256147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:48.255648 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:49.256161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:49.252532 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:49.256161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:49.252611 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:50.268146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:50.254495 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:50.268146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:50.254574 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:50.532626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:50.531293 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:55:50.532941 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:50.532922 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:55:50.658506 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:50.657246 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.257766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:51.257719 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.258080 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:51.258017 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:52.353324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.353283 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:52.373681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.373638 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:52.379144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.375405 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns"] Oct 30 01:55:52.437398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.437338 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl"] Oct 30 01:55:52.437720 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.437650 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" containerID="cri-o://ad4d26e729fc1510fb4503563c427d0cfc7b24db4b0bcb27f90cf7a10c66e3fe" gracePeriod=30 Oct 30 01:55:52.438074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.438039 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" containerID="cri-o://d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589" gracePeriod=30 Oct 30 01:55:52.726244 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.721331 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.878374 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.878321 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:55:52.972557 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:52.924201 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 01:55:53.036188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.034241 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.167582 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.167544 6275 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:55:53.169111 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.169075 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:55:53.361794 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.361591 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:53.361794 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.361647 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:53.619152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.600461 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p"] Oct 30 01:55:53.619152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.616277 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5f4e78d9-0f08-41ec-bd29-a9f172abb960" containerName="maas-api" Oct 30 01:55:53.619152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.616300 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4e78d9-0f08-41ec-bd29-a9f172abb960" containerName="maas-api" Oct 30 01:55:53.619152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.616529 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4e78d9-0f08-41ec-bd29-a9f172abb960" containerName="maas-api" Oct 30 01:55:53.619622 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.619597 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fl6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:53.619823 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.619797 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.646222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.646200 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:53.702985 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.681096 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-922b2c-serving-cert\"" Oct 30 01:55:53.802217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.798995 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:53.802217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.799108 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:53.802217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.799247 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm44t\" (UniqueName: \"kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:53.802217 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.799387 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:53.897795 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.891340 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p"] Oct 30 01:55:53.970582 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.970547 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:53.997360 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.997318 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.007182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:53.997684 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm44t\" (UniqueName: \"kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.036163 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.032911 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.036163 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:54.033639 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-serving-cert: secret "maas-api-e2e-api-a-922b2c-serving-cert" not found Oct 30 01:55:54.036163 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:54.034263 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls podName:01ffaabd-4733-4bfc-b7c7-03ae84aba138 nodeName:}" failed. No retries permitted until 2026-10-30 01:55:54.534239121 +0000 UTC m=+1582.926046751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls") pod "maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" (UID: "01ffaabd-4733-4bfc-b7c7-03ae84aba138") : secret "maas-api-e2e-api-a-922b2c-serving-cert" not found Oct 30 01:55:54.058814 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.058777 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.132143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.118723 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.196378 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.196294 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm44t\" (UniqueName: \"kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.267665 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.267630 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:55:54.267941 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.267908 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 01:55:54.578383 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.578340 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.580896 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.580874 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\") pod \"maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:54.634058 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:54.634024 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:55.070016 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:55.069920 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p"] Oct 30 01:55:55.070484 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:55:55.070457 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ffaabd_4733_4bfc_b7c7_03ae84aba138.slice/crio-2bee9254b72ee4eb49ad173c04958945df152fad2e55c24c408fb0359006756f WatchSource:0}: Error finding container 2bee9254b72ee4eb49ad173c04958945df152fad2e55c24c408fb0359006756f: Status 404 returned error can't find the container with id 2bee9254b72ee4eb49ad173c04958945df152fad2e55c24c408fb0359006756f Oct 30 01:55:55.423277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:55.423185 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" event={"ID":"01ffaabd-4733-4bfc-b7c7-03ae84aba138","Type":"ContainerStarted","Data":"2bee9254b72ee4eb49ad173c04958945df152fad2e55c24c408fb0359006756f"} Oct 30 01:55:56.301180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.299844 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:56.301180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.299923 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:56.432131 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.432079 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" event={"ID":"01ffaabd-4733-4bfc-b7c7-03ae84aba138","Type":"ContainerStarted","Data":"51acc65621b498f63db71712dfe2fd09836604b6c4b487a399478a4d39e51175"} Oct 30 01:55:56.432280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.432159 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:55:56.475388 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.475325 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" podStartSLOduration=3.063705571 podStartE2EDuration="3.475305265s" totalImagesPullingTime="411.599694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:56.473283897 +0000 UTC m=+1584.865091532" watchObservedRunningTime="2026-10-30 01:55:56.475305265 +0000 UTC m=+1584.867112899" Oct 30 01:55:56.910195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.910160 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:56.910602 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.910570 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:56.911992 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:56.911966 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 01:55:57.956015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:57.955988 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:55:58.066205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.066166 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.066539 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.066512 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.066723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.066706 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.066901 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.066883 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.067063 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.067046 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.067223 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.067204 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.067395 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.067374 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.067754 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.067731 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.069087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.069065 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket\") pod \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\" (UID: \"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5\") " Oct 30 01:55:58.070322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.070299 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.086103 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.086031 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.086616 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.086564 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.087143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.087102 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.095047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.094975 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:58.096782 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.096743 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.107374 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.107324 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:55:58.110871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.110824 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:58.116145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.111350 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx" pod "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" (UID: "bbc4b32b-8457-4756-ad2c-2e33d1ee93c5"). InnerVolumeSpecName "kube-api-access-ghsnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171650 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171689 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171706 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171723 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171736 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171750 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171762 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghsnx\" (UniqueName: \"kubernetes.io/projected/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-kube-api-access-ghsnx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171780 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.171950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.171793 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.448321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.448288 6275 generic.go:356] "Generic (PLEG): container finished" podID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerID="d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589" exitCode=0 Oct 30 01:55:58.448476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.448344 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" Oct 30 01:55:58.448476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.448361 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" event={"ID":"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5","Type":"ContainerDied","Data":"d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589"} Oct 30 01:55:58.448476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.448417 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns" event={"ID":"bbc4b32b-8457-4756-ad2c-2e33d1ee93c5","Type":"ContainerDied","Data":"5049e487d83486672bd300206f1ef5d761c8f76c62e0f2e0e34e3488e41466e4"} Oct 30 01:55:58.448476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.448442 6275 scope.go:122] "RemoveContainer" containerID="d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589" Oct 30 01:55:58.450432 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.450410 6275 generic.go:356] "Generic (PLEG): container finished" podID="67c68408-9cde-40e8-af62-8d0814b8101b" containerID="ad4d26e729fc1510fb4503563c427d0cfc7b24db4b0bcb27f90cf7a10c66e3fe" exitCode=0 Oct 30 01:55:58.450552 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.450488 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" event={"ID":"67c68408-9cde-40e8-af62-8d0814b8101b","Type":"ContainerDied","Data":"ad4d26e729fc1510fb4503563c427d0cfc7b24db4b0bcb27f90cf7a10c66e3fe"} Oct 30 01:55:58.459781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.459758 6275 scope.go:122] "RemoveContainer" containerID="0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5" Oct 30 01:55:58.472925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.472903 6275 scope.go:122] "RemoveContainer" containerID="d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589" Oct 30 01:55:58.473275 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:58.473253 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589\": container with ID starting with d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589 not found: ID does not exist" containerID="d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589" Oct 30 01:55:58.473349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.473284 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589"} err="failed to get container status \"d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589\": rpc error: code = NotFound desc = could not find container \"d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589\": container with ID starting with d0253319e0990579fbf1ba22dc84875b2fd565cd7f5dfcd767b252f9f930d589 not found: ID does not exist" Oct 30 01:55:58.473349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.473307 6275 scope.go:122] "RemoveContainer" containerID="0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5" Oct 30 01:55:58.473597 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:55:58.473578 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5\": container with ID starting with 0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5 not found: ID does not exist" containerID="0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5" Oct 30 01:55:58.473652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.473603 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5"} err="failed to get container status \"0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5\": rpc error: code = NotFound desc = could not find container \"0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5\": container with ID starting with 0c34eccc2a0f21b665c09f5ead526faa243a5f3502aefcb663d043f6e7b99ec5 not found: ID does not exist" Oct 30 01:55:58.481101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.481082 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:58.538472 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.538438 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns"] Oct 30 01:55:58.567795 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.567766 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-59nns"] Oct 30 01:55:58.578035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578010 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578044 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578063 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578096 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578133 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578165 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578434 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578249 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578434 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578329 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.578434 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578414 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.578590 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578524 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.578645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578624 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.578697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578683 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data\") pod \"67c68408-9cde-40e8-af62-8d0814b8101b\" (UID: \"67c68408-9cde-40e8-af62-8d0814b8101b\") " Oct 30 01:55:58.579027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.578683 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:58.579150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.579076 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.579356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.579333 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.579402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.579364 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.579402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.579381 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.579402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.579397 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.579495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.579411 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c68408-9cde-40e8-af62-8d0814b8101b-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.580583 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.580560 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "kube-api-access-mfwhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:58.580731 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.580708 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:55:58.580866 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.580848 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:58.580926 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.580860 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token" pod "67c68408-9cde-40e8-af62-8d0814b8101b" (UID: "67c68408-9cde-40e8-af62-8d0814b8101b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:58.680466 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.680392 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c68408-9cde-40e8-af62-8d0814b8101b-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.680466 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.680419 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.680466 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.680427 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c68408-9cde-40e8-af62-8d0814b8101b-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.680466 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:58.680435 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwhc\" (UniqueName: \"kubernetes.io/projected/67c68408-9cde-40e8-af62-8d0814b8101b-kube-api-access-mfwhc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:55:59.460421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.460381 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" Oct 30 01:55:59.460898 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.460381 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl" event={"ID":"67c68408-9cde-40e8-af62-8d0814b8101b","Type":"ContainerDied","Data":"c02cdd9b0c11afc1fb37a64a7731e133aa44df20cb376f3f4c3a7e6faab0f2b7"} Oct 30 01:55:59.460898 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.460561 6275 scope.go:122] "RemoveContainer" containerID="ad4d26e729fc1510fb4503563c427d0cfc7b24db4b0bcb27f90cf7a10c66e3fe" Oct 30 01:55:59.510324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.510292 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl"] Oct 30 01:55:59.521061 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.521030 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-rgnzl"] Oct 30 01:55:59.739598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.739507 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" path="/var/lib/kubelet/pods/67c68408-9cde-40e8-af62-8d0814b8101b/volumes" Oct 30 01:55:59.740101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:55:59.740085 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" path="/var/lib/kubelet/pods/bbc4b32b-8457-4756-ad2c-2e33d1ee93c5/volumes" Oct 30 01:56:00.260397 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:00.260362 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl"] Oct 30 01:56:00.303037 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:00.302803 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd"] Oct 30 01:56:02.445459 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:02.445431 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:56:05.261619 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.261572 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" podUID="c32b132a-05bd-48da-8cfd-d9047ca96316" containerName="payload-processing" containerID="cri-o://7f8e2380a51503bdcd5f448ce95e875fb24e9ed1294a524dacffc659f5e9611d" gracePeriod=25 Oct 30 01:56:05.303400 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.303345 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" podUID="100651e3-344b-4845-8f08-9f704d047361" containerName="payload-pre-processing" containerID="cri-o://832c6b86774cd0642db6e1231d6c2d322dc6a87bc01bb632b223150922f9ceec" gracePeriod=25 Oct 30 01:56:05.528806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.528707 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11184-f4a6d1"] Oct 30 01:56:05.530041 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530008 6275 cpu_manager.go:414] "containerMap: removing container" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" Oct 30 01:56:05.530231 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530216 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" Oct 30 01:56:05.530350 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530338 6275 cpu_manager.go:414] "containerMap: removing container" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" Oct 30 01:56:05.530432 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530421 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" Oct 30 01:56:05.530520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530510 6275 cpu_manager.go:414] "containerMap: removing container" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" Oct 30 01:56:05.530597 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530587 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" Oct 30 01:56:05.530855 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530841 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" Oct 30 01:56:05.530966 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.530956 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c68408-9cde-40e8-af62-8d0814b8101b" containerName="istio-proxy" Oct 30 01:56:05.537949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.537924 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:05.554074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.554046 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sddb\"" Oct 30 01:56:05.555826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.555788 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-f4a6d1"] Oct 30 01:56:05.560589 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.560546 6275 generic.go:356] "Generic (PLEG): container finished" podID="c32b132a-05bd-48da-8cfd-d9047ca96316" containerID="7f8e2380a51503bdcd5f448ce95e875fb24e9ed1294a524dacffc659f5e9611d" exitCode=0 Oct 30 01:56:05.560826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.560793 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" event={"ID":"c32b132a-05bd-48da-8cfd-d9047ca96316","Type":"ContainerDied","Data":"7f8e2380a51503bdcd5f448ce95e875fb24e9ed1294a524dacffc659f5e9611d"} Oct 30 01:56:05.563221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.563191 6275 generic.go:356] "Generic (PLEG): container finished" podID="100651e3-344b-4845-8f08-9f704d047361" containerID="832c6b86774cd0642db6e1231d6c2d322dc6a87bc01bb632b223150922f9ceec" exitCode=0 Oct 30 01:56:05.563313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.563263 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" event={"ID":"100651e3-344b-4845-8f08-9f704d047361","Type":"ContainerDied","Data":"832c6b86774cd0642db6e1231d6c2d322dc6a87bc01bb632b223150922f9ceec"} Oct 30 01:56:05.563918 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:05.563898 6275 logging.go:55] [core] [Channel #710 SubChannel #711]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.91:50052", ServerName: "10.134.0.91:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.91:50052: connect: connection refused" Oct 30 01:56:05.564519 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:05.564496 6275 logging.go:55] [core] [Channel #712 SubChannel #713]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.90:50052", ServerName: "10.134.0.90:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.90:50052: connect: connection refused" Oct 30 01:56:05.571114 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.571073 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9k9\" (UniqueName: \"kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9\") pod \"test-curl-11184-f4a6d1\" (UID: \"733ffac4-e3f1-4d73-a399-4aca7ed0c667\") " pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:05.678082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.673474 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9k9\" (UniqueName: \"kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9\") pod \"test-curl-11184-f4a6d1\" (UID: \"733ffac4-e3f1-4d73-a399-4aca7ed0c667\") " pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:05.695314 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.695272 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9k9\" (UniqueName: \"kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9\") pod \"test-curl-11184-f4a6d1\" (UID: \"733ffac4-e3f1-4d73-a399-4aca7ed0c667\") " pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:05.736221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.736190 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:56:05.766283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.766256 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:56:05.773965 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.773945 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume\") pod \"c32b132a-05bd-48da-8cfd-d9047ca96316\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " Oct 30 01:56:05.774154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.774114 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj\" (UniqueName: \"kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj\") pod \"c32b132a-05bd-48da-8cfd-d9047ca96316\" (UID: \"c32b132a-05bd-48da-8cfd-d9047ca96316\") " Oct 30 01:56:05.774364 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.774340 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume" pod "c32b132a-05bd-48da-8cfd-d9047ca96316" (UID: "c32b132a-05bd-48da-8cfd-d9047ca96316"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:05.774632 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.774615 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32b132a-05bd-48da-8cfd-d9047ca96316-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:05.776303 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.776280 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj" pod "c32b132a-05bd-48da-8cfd-d9047ca96316" (UID: "c32b132a-05bd-48da-8cfd-d9047ca96316"). InnerVolumeSpecName "kube-api-access-zqzkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:05.855216 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.855111 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:05.876344 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.876315 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt\" (UniqueName: \"kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt\") pod \"100651e3-344b-4845-8f08-9f704d047361\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " Oct 30 01:56:05.876545 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.876526 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume\") pod \"100651e3-344b-4845-8f08-9f704d047361\" (UID: \"100651e3-344b-4845-8f08-9f704d047361\") " Oct 30 01:56:05.876882 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.876857 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume" pod "100651e3-344b-4845-8f08-9f704d047361" (UID: "100651e3-344b-4845-8f08-9f704d047361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:05.877184 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.877166 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzkj\" (UniqueName: \"kubernetes.io/projected/c32b132a-05bd-48da-8cfd-d9047ca96316-kube-api-access-zqzkj\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:05.877244 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.877189 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100651e3-344b-4845-8f08-9f704d047361-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:05.878308 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.878280 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt" pod "100651e3-344b-4845-8f08-9f704d047361" (UID: "100651e3-344b-4845-8f08-9f704d047361"). InnerVolumeSpecName "kube-api-access-rcgpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:05.979139 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:05.979090 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgpt\" (UniqueName: \"kubernetes.io/projected/100651e3-344b-4845-8f08-9f704d047361-kube-api-access-rcgpt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:06.165485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.165402 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-f4a6d1"] Oct 30 01:56:06.170409 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:06.170364 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod733ffac4_e3f1_4d73_a399_4aca7ed0c667.slice/crio-792bbe0e8b320019f845b80f8873c2270d649d34424c8ed274f69c9c9c904f24 WatchSource:0}: Error finding container 792bbe0e8b320019f845b80f8873c2270d649d34424c8ed274f69c9c9c904f24: Status 404 returned error can't find the container with id 792bbe0e8b320019f845b80f8873c2270d649d34424c8ed274f69c9c9c904f24 Oct 30 01:56:06.563626 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.563585 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" podUID="100651e3-344b-4845-8f08-9f704d047361" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.91:50052\" within 1s: context deadline exceeded" Oct 30 01:56:06.564767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.564732 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" podUID="c32b132a-05bd-48da-8cfd-d9047ca96316" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.90:50052\" within 1s: context deadline exceeded" Oct 30 01:56:06.571331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.571307 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" Oct 30 01:56:06.571331 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.571314 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd" event={"ID":"100651e3-344b-4845-8f08-9f704d047361","Type":"ContainerDied","Data":"e0093540e0c425a897f18126a0cfeb5d81865dd547bf96f40abef925cecc571f"} Oct 30 01:56:06.571538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.571408 6275 scope.go:122] "RemoveContainer" containerID="832c6b86774cd0642db6e1231d6c2d322dc6a87bc01bb632b223150922f9ceec" Oct 30 01:56:06.572797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.572506 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-f4a6d1" event={"ID":"733ffac4-e3f1-4d73-a399-4aca7ed0c667","Type":"ContainerStarted","Data":"792bbe0e8b320019f845b80f8873c2270d649d34424c8ed274f69c9c9c904f24"} Oct 30 01:56:06.574040 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.574017 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" event={"ID":"c32b132a-05bd-48da-8cfd-d9047ca96316","Type":"ContainerDied","Data":"5c9c9ee64d97d536e68f445ea591d3514b3d312ad48158c9a101515489d18c97"} Oct 30 01:56:06.574170 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.574068 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl" Oct 30 01:56:06.583394 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.583296 6275 scope.go:122] "RemoveContainer" containerID="7f8e2380a51503bdcd5f448ce95e875fb24e9ed1294a524dacffc659f5e9611d" Oct 30 01:56:06.621898 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.621871 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd"] Oct 30 01:56:06.631384 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.631357 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c2993e-7879c4999f-tsbxd"] Oct 30 01:56:06.661309 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.661278 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl"] Oct 30 01:56:06.666991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:06.666965 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c2993e-6965595d66-mwsdl"] Oct 30 01:56:07.742302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:07.742272 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100651e3-344b-4845-8f08-9f704d047361" path="/var/lib/kubelet/pods/100651e3-344b-4845-8f08-9f704d047361/volumes" Oct 30 01:56:07.742698 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:07.742669 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32b132a-05bd-48da-8cfd-d9047ca96316" path="/var/lib/kubelet/pods/c32b132a-05bd-48da-8cfd-d9047ca96316/volumes" Oct 30 01:56:11.366237 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.366201 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4"] Oct 30 01:56:11.367051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367031 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c32b132a-05bd-48da-8cfd-d9047ca96316" containerName="payload-processing" Oct 30 01:56:11.367115 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367053 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32b132a-05bd-48da-8cfd-d9047ca96316" containerName="payload-processing" Oct 30 01:56:11.367179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367139 6275 cpu_manager.go:414] "containerMap: removing container" podUID="100651e3-344b-4845-8f08-9f704d047361" containerName="payload-pre-processing" Oct 30 01:56:11.367179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367147 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100651e3-344b-4845-8f08-9f704d047361" containerName="payload-pre-processing" Oct 30 01:56:11.367272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367262 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="100651e3-344b-4845-8f08-9f704d047361" containerName="payload-pre-processing" Oct 30 01:56:11.367307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367276 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc4b32b-8457-4756-ad2c-2e33d1ee93c5" containerName="istio-proxy" Oct 30 01:56:11.367307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.367284 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32b132a-05bd-48da-8cfd-d9047ca96316" containerName="payload-processing" Oct 30 01:56:11.372622 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.372600 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.391512 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.391490 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c93367-openshift-default-dockercfg-vcfsd\"" Oct 30 01:56:11.407498 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.407463 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4"] Oct 30 01:56:11.441693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.441659 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.441871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.441704 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.441871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.441803 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.441871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.441857 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.442066 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.441921 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fkp\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.442066 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.442026 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.442265 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.442073 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.442364 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.442260 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.442440 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.442413 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.546084 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.546037 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.547805 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.547776 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fkp\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.548485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.548463 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.549001 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.548983 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.549614 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.549596 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.550285 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.550256 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.551045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.551022 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.565849 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.565820 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.566833 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.566813 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.567411 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.549462 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.567545 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.566729 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.567634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.550041 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.567714 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.567231 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.567796 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.550712 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.587296 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.587261 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fkp\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.620194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.617698 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.620194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.617735 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.646142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.641836 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo\") pod \"e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:11.698841 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:11.695429 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:12.099662 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.099629 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4"] Oct 30 01:56:12.103432 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:12.103408 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783ef77e_9b40_4eb0_8062_e35f4446011c.slice/crio-72cebe001c21d2e8e46eb3e93d650c8f1b2727c09998eebbad5270b537748a2c WatchSource:0}: Error finding container 72cebe001c21d2e8e46eb3e93d650c8f1b2727c09998eebbad5270b537748a2c: Status 404 returned error can't find the container with id 72cebe001c21d2e8e46eb3e93d650c8f1b2727c09998eebbad5270b537748a2c Oct 30 01:56:12.106009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.105976 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:12.106081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.106048 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:12.106081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.106076 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:12.683146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.681769 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" event={"ID":"783ef77e-9b40-4eb0-8062-e35f4446011c","Type":"ContainerStarted","Data":"473665835c2b288b68e1b19f4889564031d9bed0b8b6a8c5e2350288926a046f"} Oct 30 01:56:12.683146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.681841 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" event={"ID":"783ef77e-9b40-4eb0-8062-e35f4446011c","Type":"ContainerStarted","Data":"72cebe001c21d2e8e46eb3e93d650c8f1b2727c09998eebbad5270b537748a2c"} Oct 30 01:56:12.696232 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.696187 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:12.754634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:12.754584 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podStartSLOduration=1.754567537 podStartE2EDuration="1.754567537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:12.738832673 +0000 UTC m=+1601.130640307" watchObservedRunningTime="2026-10-30 01:56:12.754567537 +0000 UTC m=+1601.146375169" Oct 30 01:56:13.703853 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:13.703817 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:14.698027 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:14.697983 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-f4a6d1" event={"ID":"733ffac4-e3f1-4d73-a399-4aca7ed0c667","Type":"ContainerStarted","Data":"05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f"} Oct 30 01:56:14.698370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:14.698342 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:14.699537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:14.699517 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:14.733847 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:14.733797 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11184-f4a6d1" podStartSLOduration=1.696534016 podStartE2EDuration="9.733782828s" totalImagesPullingTime="8.037248812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:14.733491989 +0000 UTC m=+1603.125299624" watchObservedRunningTime="2026-10-30 01:56:14.733782828 +0000 UTC m=+1603.125590458" Oct 30 01:56:15.610451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:15.610416 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11184-f4a6d1"] Oct 30 01:56:15.614001 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:15.613972 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11184-f4a6d1"] Oct 30 01:56:16.372142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.372098 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z"] Oct 30 01:56:16.372632 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.372356 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" podUID="db4e0f1b-f85c-412f-9e8d-c26b38557344" containerName="maas-api" containerID="cri-o://b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc" gracePeriod=30 Oct 30 01:56:16.765145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.748597 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:56:16.772147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.768400 6275 generic.go:356] "Generic (PLEG): container finished" podID="db4e0f1b-f85c-412f-9e8d-c26b38557344" containerID="b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc" exitCode=0 Oct 30 01:56:16.772147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.769584 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" event={"ID":"db4e0f1b-f85c-412f-9e8d-c26b38557344","Type":"ContainerDied","Data":"b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc"} Oct 30 01:56:16.772147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.769637 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" event={"ID":"db4e0f1b-f85c-412f-9e8d-c26b38557344","Type":"ContainerDied","Data":"e8574bbcb1c01abdff38a1ee8db79e51b7bd3236bb7c8afcd458ad05050aeb45"} Oct 30 01:56:16.772147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.769669 6275 scope.go:122] "RemoveContainer" containerID="b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc" Oct 30 01:56:16.772147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.769883 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11184-f4a6d1" podUID="733ffac4-e3f1-4d73-a399-4aca7ed0c667" containerName="test-curl-11184-f4a6d1" containerID="cri-o://05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f" gracePeriod=2 Oct 30 01:56:16.816398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.810343 6275 scope.go:122] "RemoveContainer" containerID="b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc" Oct 30 01:56:16.833146 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:16.831780 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc\": container with ID starting with b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc not found: ID does not exist" containerID="b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc" Oct 30 01:56:16.833146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.831832 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc"} err="failed to get container status \"b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc\": rpc error: code = NotFound desc = could not find container \"b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc\": container with ID starting with b80a765ff5821841aee39e0464334db39a5cc02232fda342cc19532e1ad71afc not found: ID does not exist" Oct 30 01:56:16.879726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.871652 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls\") pod \"db4e0f1b-f85c-412f-9e8d-c26b38557344\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " Oct 30 01:56:16.879726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.871781 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs\") pod \"db4e0f1b-f85c-412f-9e8d-c26b38557344\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " Oct 30 01:56:16.879726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.871848 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k\" (UniqueName: \"kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k\") pod \"db4e0f1b-f85c-412f-9e8d-c26b38557344\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " Oct 30 01:56:16.879726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.872091 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca\") pod \"db4e0f1b-f85c-412f-9e8d-c26b38557344\" (UID: \"db4e0f1b-f85c-412f-9e8d-c26b38557344\") " Oct 30 01:56:16.936198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.933664 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls" pod "db4e0f1b-f85c-412f-9e8d-c26b38557344" (UID: "db4e0f1b-f85c-412f-9e8d-c26b38557344"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:16.962189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.953336 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca" pod "db4e0f1b-f85c-412f-9e8d-c26b38557344" (UID: "db4e0f1b-f85c-412f-9e8d-c26b38557344"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:17.000409 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.979842 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e0f1b-f85c-412f-9e8d-c26b38557344-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:17.000409 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:16.979883 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:17.059276 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.056783 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs" pod "db4e0f1b-f85c-412f-9e8d-c26b38557344" (UID: "db4e0f1b-f85c-412f-9e8d-c26b38557344"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:17.059276 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.056914 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k" pod "db4e0f1b-f85c-412f-9e8d-c26b38557344" (UID: "db4e0f1b-f85c-412f-9e8d-c26b38557344"). InnerVolumeSpecName "kube-api-access-49f4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:17.081691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.081628 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4e0f1b-f85c-412f-9e8d-c26b38557344-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:17.081691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.081662 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f4k\" (UniqueName: \"kubernetes.io/projected/db4e0f1b-f85c-412f-9e8d-c26b38557344-kube-api-access-49f4k\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.109282 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr"] Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.110308 6275 cpu_manager.go:414] "containerMap: removing container" podUID="db4e0f1b-f85c-412f-9e8d-c26b38557344" containerName="maas-api" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.110330 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4e0f1b-f85c-412f-9e8d-c26b38557344" containerName="maas-api" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.110408 6275 cpu_manager.go:414] "containerMap: removing container" podUID="733ffac4-e3f1-4d73-a399-4aca7ed0c667" containerName="test-curl-11184-f4a6d1" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.110416 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733ffac4-e3f1-4d73-a399-4aca7ed0c667" containerName="test-curl-11184-f4a6d1" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.110551 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4e0f1b-f85c-412f-9e8d-c26b38557344" containerName="maas-api" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.110574 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="733ffac4-e3f1-4d73-a399-4aca7ed0c667" containerName="test-curl-11184-f4a6d1" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.116144 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.152207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.150331 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-922b2c-openshift-default-dockercfg-p78gv\"" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.185780 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.185845 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.185910 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.186012 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.186115 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.186171 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.186227 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.186297 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.190316 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.186327 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrjb\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.214247 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.210205 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k"] Oct 30 01:56:17.238522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.219837 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.238522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.223993 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr"] Oct 30 01:56:17.238522 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:17.229592 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:56:17.239258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.239210 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k"] Oct 30 01:56:17.287398 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287363 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287414 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287438 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.287585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287455 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.287585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287494 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287585 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287513 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrjb\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287611 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287641 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287731 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287734 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287805 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287819 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287828 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdmf\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.287837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287840 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.288245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287903 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.288245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287934 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.288245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287971 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.288245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.287998 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.288245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.288054 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.288245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.288165 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.288450 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.288250 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.288450 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.288257 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.288855 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.288839 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.290083 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.290060 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.290385 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.290370 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.324544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.324499 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.325196 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.325172 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrjb\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.390052 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390017 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390066 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390149 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390225 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdmf\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390258 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390307 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390348 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390567 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390583 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390608 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.390908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390900 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.391109 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.390905 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.391335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.391308 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.391417 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.391338 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.392820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.392796 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.393508 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.393487 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.419278 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.419253 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.419439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.419417 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdmf\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.532770 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.532732 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:17.558360 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.558327 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:17.775772 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.775737 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z" Oct 30 01:56:17.816150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.815272 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv"] Oct 30 01:56:17.824764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.824726 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.866829 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.866792 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5"] Oct 30 01:56:17.872188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.872163 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv"] Oct 30 01:56:17.872362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.872298 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:17.880275 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.880241 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b2c11f-openshift-default-dockercfg-glvhd\"" Oct 30 01:56:17.883654 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.883627 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z"] Oct 30 01:56:17.892362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.892337 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c2993e-78d9f8c7d-6894z"] Oct 30 01:56:17.898064 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898037 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjq45\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898220 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898306 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898436 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898417 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898464 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898589 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898568 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898639 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898716 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898682 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.898872 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.898852 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:17.910274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.910210 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c93367\"" Oct 30 01:56:17.912463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.912437 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5"] Oct 30 01:56:17.931064 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.931035 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4"] Oct 30 01:56:17.937152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.936464 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:17.941811 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.941788 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k"] Oct 30 01:56:17.945268 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.945242 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr"] Oct 30 01:56:17.945441 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:17.945421 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c71dd7_6c92_4f06_9501_7247cf7673a1.slice/crio-e92bb5f887a9d0a01e1c660183686dd6056e52d7e1cf1081abee6e6da0bda980 WatchSource:0}: Error finding container e92bb5f887a9d0a01e1c660183686dd6056e52d7e1cf1081abee6e6da0bda980: Status 404 returned error can't find the container with id e92bb5f887a9d0a01e1c660183686dd6056e52d7e1cf1081abee6e6da0bda980 Oct 30 01:56:17.947229 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:17.947205 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c718437_d257_46ba_a3d4_14f6aeb5d40e.slice/crio-0d1e558cd79d659a872161b3de3c8c7495d3480b56bf52fa9b7e1afbf6d64291 WatchSource:0}: Error finding container 0d1e558cd79d659a872161b3de3c8c7495d3480b56bf52fa9b7e1afbf6d64291: Status 404 returned error can't find the container with id 0d1e558cd79d659a872161b3de3c8c7495d3480b56bf52fa9b7e1afbf6d64291 Oct 30 01:56:17.948968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.948929 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:17.949039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.949021 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:17.949087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.949071 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:17.949987 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.949808 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:17.949987 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.949861 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:17.949987 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.949890 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:17.960839 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.960810 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c93367-dockercfg-6569c\"" Oct 30 01:56:17.962465 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:17.962440 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4"] Oct 30 01:56:18.000832 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.000800 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.000972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.000866 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.000972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.000900 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.000972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.000930 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.001188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.000980 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.001188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001078 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.001302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001192 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjq45\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.001302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001226 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.001302 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001236 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.001481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001334 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.001481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001393 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrjn\" (UniqueName: \"kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn\") pod \"payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.001481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001427 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.002042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001756 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zwg\" (UniqueName: \"kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg\") pod \"payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.002042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.001806 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.002042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.002006 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.002479 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.002457 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.002799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.002775 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.003032 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.003012 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.005754 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.005727 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.008229 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.008204 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.122146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.084998 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjq45\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.122146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.085694 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token\") pod \"e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.122146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.110376 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrjn\" (UniqueName: \"kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn\") pod \"payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.122146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.110482 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zwg\" (UniqueName: \"kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg\") pod \"payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.122146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.110607 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.122146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.110772 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.124696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.124663 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.144307 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.112113 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.144956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.144693 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:18.224337 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.223906 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zwg\" (UniqueName: \"kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg\") pod \"payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.246712 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.245853 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrjn\" (UniqueName: \"kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn\") pod \"payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.257362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.257321 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:18.495250 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.494837 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:18.788984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.788948 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" event={"ID":"4c718437-d257-46ba-a3d4-14f6aeb5d40e","Type":"ContainerStarted","Data":"4a7dd2109f129b573030338abcbacd0301b695b73e70e255058b734eacd6616b"} Oct 30 01:56:18.793808 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.790999 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" event={"ID":"4c718437-d257-46ba-a3d4-14f6aeb5d40e","Type":"ContainerStarted","Data":"0d1e558cd79d659a872161b3de3c8c7495d3480b56bf52fa9b7e1afbf6d64291"} Oct 30 01:56:18.793888 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.793841 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" event={"ID":"62c71dd7-6c92-4f06-9501-7247cf7673a1","Type":"ContainerStarted","Data":"96af87fb7f78b6935beb17d479823163ddf84ee5c2d3a7120831bab0740e3896"} Oct 30 01:56:18.793948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.793897 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv"] Oct 30 01:56:18.793948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.793934 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" event={"ID":"62c71dd7-6c92-4f06-9501-7247cf7673a1","Type":"ContainerStarted","Data":"e92bb5f887a9d0a01e1c660183686dd6056e52d7e1cf1081abee6e6da0bda980"} Oct 30 01:56:18.827301 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.827264 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:18.827408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.827332 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:18.827408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.827362 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:18.848745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.848702 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podStartSLOduration=2.848686559 podStartE2EDuration="2.848686559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:18.843046165 +0000 UTC m=+1607.234853800" watchObservedRunningTime="2026-10-30 01:56:18.848686559 +0000 UTC m=+1607.240494393" Oct 30 01:56:18.856380 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.856351 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4"] Oct 30 01:56:18.899989 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.899934 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podStartSLOduration=2.89991541 podStartE2EDuration="2.89991541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:18.891541995 +0000 UTC m=+1607.283349647" watchObservedRunningTime="2026-10-30 01:56:18.89991541 +0000 UTC m=+1607.291723044" Oct 30 01:56:19.009935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:18.990240 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5"] Oct 30 01:56:19.237903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.237870 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:19.313099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.313009 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9\" (UniqueName: \"kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9\") pod \"733ffac4-e3f1-4d73-a399-4aca7ed0c667\" (UID: \"733ffac4-e3f1-4d73-a399-4aca7ed0c667\") " Oct 30 01:56:19.315376 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.315345 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9" pod "733ffac4-e3f1-4d73-a399-4aca7ed0c667" (UID: "733ffac4-e3f1-4d73-a399-4aca7ed0c667"). InnerVolumeSpecName "kube-api-access-xn9k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:19.415443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.415412 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9k9\" (UniqueName: \"kubernetes.io/projected/733ffac4-e3f1-4d73-a399-4aca7ed0c667-kube-api-access-xn9k9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:19.534081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.534046 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:19.534819 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.534795 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:56:19.534936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.534847 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:56:19.565206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.558489 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:19.594694 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.594658 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:56:19.595005 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.594956 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 01:56:19.794186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.791159 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733ffac4-e3f1-4d73-a399-4aca7ed0c667" path="/var/lib/kubelet/pods/733ffac4-e3f1-4d73-a399-4aca7ed0c667/volumes" Oct 30 01:56:19.794186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.791969 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4e0f1b-f85c-412f-9e8d-c26b38557344" path="/var/lib/kubelet/pods/db4e0f1b-f85c-412f-9e8d-c26b38557344/volumes" Oct 30 01:56:19.808581 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.808545 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" event={"ID":"9176efeb-3e9e-468b-bd0f-e5ee725a52f7","Type":"ContainerStarted","Data":"320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5"} Oct 30 01:56:19.808690 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.808617 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" event={"ID":"9176efeb-3e9e-468b-bd0f-e5ee725a52f7","Type":"ContainerStarted","Data":"0d2c8762a6d87d88d706d5821c451a91618682442f56157fd0ef1678079a67d5"} Oct 30 01:56:19.809837 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.809814 6275 generic.go:356] "Generic (PLEG): container finished" podID="733ffac4-e3f1-4d73-a399-4aca7ed0c667" containerID="05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f" exitCode=137 Oct 30 01:56:19.809919 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.809906 6275 scope.go:122] "RemoveContainer" containerID="05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f" Oct 30 01:56:19.810050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.810040 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-f4a6d1" Oct 30 01:56:19.812460 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.812429 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" event={"ID":"9b9c3dee-8334-40ed-a4b5-3c185a8573ae","Type":"ContainerStarted","Data":"d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69"} Oct 30 01:56:19.812598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.812487 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" event={"ID":"9b9c3dee-8334-40ed-a4b5-3c185a8573ae","Type":"ContainerStarted","Data":"7a13bcbfe3825c8e99ba82f82236763d997bcc10d66f092a68449712bb4b92d0"} Oct 30 01:56:19.812598 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.812520 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:19.814071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.814045 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" event={"ID":"f6837615-dbb6-431c-9052-070cd52a5772","Type":"ContainerStarted","Data":"fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518"} Oct 30 01:56:19.814180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.814088 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" event={"ID":"f6837615-dbb6-431c-9052-070cd52a5772","Type":"ContainerStarted","Data":"3e48d8c6ac26e5b501ae96aef43f9413eec4db12eff00374027c8d6e7633b96a"} Oct 30 01:56:19.814180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.814112 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:19.820932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.820912 6275 scope.go:122] "RemoveContainer" containerID="05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f" Oct 30 01:56:19.821228 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:19.821206 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f\": container with ID starting with 05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f not found: ID does not exist" containerID="05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f" Oct 30 01:56:19.821273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.821235 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f"} err="failed to get container status \"05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f\": rpc error: code = NotFound desc = could not find container \"05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f\": container with ID starting with 05e1715c816cbecd01954ab026dfd5fa313d4123f159335466b06093a2a0392f not found: ID does not exist" Oct 30 01:56:19.913591 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.913532 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podStartSLOduration=2.913513165 podStartE2EDuration="2.913513165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:19.859509115 +0000 UTC m=+1608.251316761" watchObservedRunningTime="2026-10-30 01:56:19.913513165 +0000 UTC m=+1608.305320800" Oct 30 01:56:19.916303 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.916254 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" podStartSLOduration=2.916245524 podStartE2EDuration="2.916245524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:19.911108164 +0000 UTC m=+1608.302915811" watchObservedRunningTime="2026-10-30 01:56:19.916245524 +0000 UTC m=+1608.308053201" Oct 30 01:56:19.971145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:19.970618 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" podStartSLOduration=2.970596992 podStartE2EDuration="2.970596992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:19.964628978 +0000 UTC m=+1608.356436606" watchObservedRunningTime="2026-10-30 01:56:19.970596992 +0000 UTC m=+1608.362404626" Oct 30 01:56:20.152773 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.152676 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:20.162484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.162451 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:20.567148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.563220 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:56:20.567148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.563283 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:56:20.567148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.563696 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:56:20.567148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.563736 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 01:56:20.824523 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.823918 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:20.825825 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:20.825800 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:56:21.533764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:21.533728 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:56:21.533928 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:21.533787 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:56:21.559649 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:21.559611 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:56:21.559950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:21.559902 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 01:56:22.533327 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:22.533251 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:56:22.533709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:22.533317 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:56:22.566775 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:22.566742 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:56:22.567047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:22.567014 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 01:56:23.533807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:23.533641 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:56:23.533807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:23.533702 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:56:23.559014 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:23.558979 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:56:23.559263 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:23.559236 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 01:56:24.565144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.564769 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:56:24.565144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.564831 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 01:56:24.602155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.586755 6275 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:56:24.602155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.586830 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:56:24.870194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.869317 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf"] Oct 30 01:56:24.938637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.934423 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.011143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:24.999516 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b2c11f-dockercfg-6fjt8\"" Oct 30 01:56:25.056201 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.039203 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b2c11f\"" Oct 30 01:56:25.056201 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.044649 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdb9\" (UniqueName: \"kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9\") pod \"payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.056201 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.044743 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.147883 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.147778 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf"] Oct 30 01:56:25.159390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.157530 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdb9\" (UniqueName: \"kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9\") pod \"payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.159390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.157615 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.159390 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.158679 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.216189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.216094 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdb9\" (UniqueName: \"kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9\") pod \"payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.484476 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.484385 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:25.588843 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.588813 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:25.593346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.592351 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:25.593346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.592800 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:25.593909 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.593871 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:25.597071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.597044 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:56:25.598812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.598789 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:56:25.884601 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:25.884573 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf"] Oct 30 01:56:25.886565 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:56:25.886530 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aab290_ff40_45d2_9592_3d9b0f5fd56f.slice/crio-3f5a0adafdce9206f0983e2807b534d2c79dd78c5f209e177be92452d098ca3d WatchSource:0}: Error finding container 3f5a0adafdce9206f0983e2807b534d2c79dd78c5f209e177be92452d098ca3d: Status 404 returned error can't find the container with id 3f5a0adafdce9206f0983e2807b534d2c79dd78c5f209e177be92452d098ca3d Oct 30 01:56:26.141310 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:26.141235 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" event={"ID":"c5aab290-ff40-45d2-9592-3d9b0f5fd56f","Type":"ContainerStarted","Data":"1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede"} Oct 30 01:56:26.141310 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:26.141290 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" event={"ID":"c5aab290-ff40-45d2-9592-3d9b0f5fd56f","Type":"ContainerStarted","Data":"3f5a0adafdce9206f0983e2807b534d2c79dd78c5f209e177be92452d098ca3d"} Oct 30 01:56:26.141724 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:26.141706 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:26.247154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:26.232498 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" podStartSLOduration=2.232476753 podStartE2EDuration="2.232476753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:26.206035185 +0000 UTC m=+1614.597842834" watchObservedRunningTime="2026-10-30 01:56:26.232476753 +0000 UTC m=+1614.624284391" Oct 30 01:56:26.782889 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:26.782846 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4"] Oct 30 01:56:27.184975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:27.172111 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6"] Oct 30 01:56:27.184975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:27.172381 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" containerID="cri-o://1b7f9b557d3c3e9aaf15d0d35ca77fb0c1ec93d866214bbe126dc4c96f55ca69" gracePeriod=30 Oct 30 01:56:30.225049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:30.225011 6275 patch_prober.go:29] interesting pod/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 01:56:30.225458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:30.225081 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:56:30.865518 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:30.862312 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:56:30.865518 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:30.863737 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:56:31.784101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:31.784062 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerName="payload-processing" containerID="cri-o://a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9" gracePeriod=25 Oct 30 01:56:32.335046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.334496 6275 generic.go:356] "Generic (PLEG): container finished" podID="03107880-070a-4f45-b875-b1cd7468fe2a" containerID="1b7f9b557d3c3e9aaf15d0d35ca77fb0c1ec93d866214bbe126dc4c96f55ca69" exitCode=0 Oct 30 01:56:32.335046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.334787 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" event={"ID":"03107880-070a-4f45-b875-b1cd7468fe2a","Type":"ContainerDied","Data":"1b7f9b557d3c3e9aaf15d0d35ca77fb0c1ec93d866214bbe126dc4c96f55ca69"} Oct 30 01:56:32.335046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.334881 6275 scope.go:122] "RemoveContainer" containerID="7a45614bf5d39401946a19d64be45dde6aebf4651aad8ea1085b0caf27f7aed2" Oct 30 01:56:32.381324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.381286 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km"] Oct 30 01:56:32.448781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.448744 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km"] Oct 30 01:56:32.448984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.448924 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.484644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.482865 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqbb\" (UniqueName: \"kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.484644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.482946 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.484644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.483177 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.484644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.483321 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.489258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.489229 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c93367-serving-cert\"" Oct 30 01:56:32.603164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.585794 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.603164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.585932 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqbb\" (UniqueName: \"kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.603164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.586007 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.603164 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.586233 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.623147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.604610 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.675146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.663828 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqbb\" (UniqueName: \"kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.807997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.766607 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.838053 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.836696 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls\") pod \"maas-api-e2e-api-a-c93367-8794dc4b7-nk7km\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:32.842661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:32.842400 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:33.183065 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.181640 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:56:33.284482 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284428 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.284689 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284551 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.284689 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284612 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.284689 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284655 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.284880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284709 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.284880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284761 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.284880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284840 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.285047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284907 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.285047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284967 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:33.285047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.284984 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:33.285238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285159 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token\") pod \"03107880-070a-4f45-b875-b1cd7468fe2a\" (UID: \"03107880-070a-4f45-b875-b1cd7468fe2a\") " Oct 30 01:56:33.285238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285215 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:33.285356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285320 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:33.285963 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285882 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.285963 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285909 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.285963 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285926 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.285963 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.285940 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03107880-070a-4f45-b875-b1cd7468fe2a-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.286502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.286481 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:33.288655 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.288618 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:33.289184 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.289107 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:33.289478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.289446 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "kube-api-access-jhtdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:33.289730 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.289703 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo" pod "03107880-070a-4f45-b875-b1cd7468fe2a" (UID: "03107880-070a-4f45-b875-b1cd7468fe2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:56:33.344793 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.344705 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" Oct 30 01:56:33.344793 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.344718 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6" event={"ID":"03107880-070a-4f45-b875-b1cd7468fe2a","Type":"ContainerDied","Data":"c161d07121644a040052efac86fd042954eb188070e63c2a5d16eff21666a147"} Oct 30 01:56:33.345021 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.344797 6275 scope.go:122] "RemoveContainer" containerID="1b7f9b557d3c3e9aaf15d0d35ca77fb0c1ec93d866214bbe126dc4c96f55ca69" Oct 30 01:56:33.387722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.387693 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtdf\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-kube-api-access-jhtdf\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.387722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.387721 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03107880-070a-4f45-b875-b1cd7468fe2a-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.387958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.387734 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03107880-070a-4f45-b875-b1cd7468fe2a-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.387958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.387747 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.387958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.387761 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03107880-070a-4f45-b875-b1cd7468fe2a-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:33.413324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.413187 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km"] Oct 30 01:56:33.426803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.426760 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6"] Oct 30 01:56:33.454254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.454215 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c2993e-openshift-default-7cb8894cb9-2lvk6"] Oct 30 01:56:33.764641 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:33.764592 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" path="/var/lib/kubelet/pods/03107880-070a-4f45-b875-b1cd7468fe2a/volumes" Oct 30 01:56:34.418205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:34.417622 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" event={"ID":"dc7fe934-149b-4b99-b5a7-d33262bfc32d","Type":"ContainerStarted","Data":"28b3582d8f3b1313ed190ae5be195bf8e33ef48aeff7e34862516fbf1484e764"} Oct 30 01:56:34.418205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:34.417692 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" event={"ID":"dc7fe934-149b-4b99-b5a7-d33262bfc32d","Type":"ContainerStarted","Data":"626c113db066dae766ccf25490a792016c2c893152ce5e98c25d95c2155c278d"} Oct 30 01:56:34.424169 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:34.421365 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:34.499144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:34.498294 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" podStartSLOduration=1.9498168310000001 podStartE2EDuration="2.498272942s" totalImagesPullingTime="548.456111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:34.49562047 +0000 UTC m=+1622.887428104" watchObservedRunningTime="2026-10-30 01:56:34.498272942 +0000 UTC m=+1622.890080576" Oct 30 01:56:35.110149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:35.107095 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:56:35.110149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:35.107176 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:37.191661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:37.191634 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:56:38.248161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.248114 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d"] Oct 30 01:56:38.249158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.249131 6275 cpu_manager.go:414] "containerMap: removing container" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" Oct 30 01:56:38.249158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.249151 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" Oct 30 01:56:38.249158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.249165 6275 cpu_manager.go:414] "containerMap: removing container" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" Oct 30 01:56:38.249424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.249171 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" Oct 30 01:56:38.249424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.249316 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" Oct 30 01:56:38.249424 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.249322 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="03107880-070a-4f45-b875-b1cd7468fe2a" containerName="istio-proxy" Oct 30 01:56:38.255917 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.255894 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.270328 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.270303 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b2c11f-serving-cert\"" Oct 30 01:56:38.286106 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.286076 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d"] Oct 30 01:56:38.375908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.375868 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.376093 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.376061 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.376205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.376189 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.376324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.376312 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5tn\" (UniqueName: \"kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.479619 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.479578 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.480112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.480086 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.480366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.480347 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.480652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.480632 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5tn\" (UniqueName: \"kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.482219 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:38.482197 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-serving-cert: secret "maas-api-e2e-api-b-b2c11f-serving-cert" not found Oct 30 01:56:38.482541 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:38.482524 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls podName:9f553128-bc31-47f7-b454-89624973bfa9 nodeName:}" failed. No retries permitted until 2026-10-30 01:56:38.982472984 +0000 UTC m=+1627.374280611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls") pod "maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" (UID: "9f553128-bc31-47f7-b454-89624973bfa9") : secret "maas-api-e2e-api-b-b2c11f-serving-cert" not found Oct 30 01:56:38.483917 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.483895 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.493799 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.493769 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.510317 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.510292 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5tn\" (UniqueName: \"kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.987886 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.987784 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:38.990216 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:38.990190 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:39.182846 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:39.181797 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:39.577597 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:39.577043 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d"] Oct 30 01:56:40.459992 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:40.458304 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:56:40.605208 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:40.605115 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" event={"ID":"9f553128-bc31-47f7-b454-89624973bfa9","Type":"ContainerStarted","Data":"0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222"} Oct 30 01:56:40.605574 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:40.605242 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:40.605574 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:40.605269 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" event={"ID":"9f553128-bc31-47f7-b454-89624973bfa9","Type":"ContainerStarted","Data":"f1657dc8e4e16ba65ce63af5d9a526e6b8756e556b4f3ca7062e545be0ca6b25"} Oct 30 01:56:40.656713 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:40.656649 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" podStartSLOduration=2.308633322 podStartE2EDuration="2.656634864s" totalImagesPullingTime="348.001542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:40.654918701 +0000 UTC m=+1629.046726335" watchObservedRunningTime="2026-10-30 01:56:40.656634864 +0000 UTC m=+1629.048442496" Oct 30 01:56:41.575196 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:41.575155 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:56:46.617663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:46.617634 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:56:46.706936 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:46.706897 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:56:51.573681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:51.573636 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:56:52.118186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.118158 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:56:52.228033 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.227948 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv\" (UniqueName: \"kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv\") pod \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " Oct 30 01:56:52.228223 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.228205 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume\") pod \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\" (UID: \"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5\") " Oct 30 01:56:52.228668 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.228634 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume" pod "8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" (UID: "8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:52.228807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.228791 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:52.232289 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.232262 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv" pod "8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" (UID: "8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5"). InnerVolumeSpecName "kube-api-access-59tsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:52.330353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.330314 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tsv\" (UniqueName: \"kubernetes.io/projected/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5-kube-api-access-59tsv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:56:52.787088 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.787048 6275 generic.go:356] "Generic (PLEG): container finished" podID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerID="a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9" exitCode=0 Oct 30 01:56:52.787579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.787092 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" Oct 30 01:56:52.787579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.787163 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" event={"ID":"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5","Type":"ContainerDied","Data":"a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9"} Oct 30 01:56:52.787579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.787223 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4" event={"ID":"8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5","Type":"ContainerDied","Data":"43d7f4a2065f51ed97cc1311c31e442615d28e3460a11be3ded8821b70d68721"} Oct 30 01:56:52.787579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.787260 6275 scope.go:122] "RemoveContainer" containerID="a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9" Oct 30 01:56:52.797760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.797744 6275 scope.go:122] "RemoveContainer" containerID="a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9" Oct 30 01:56:52.798042 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:56:52.798019 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9\": container with ID starting with a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9 not found: ID does not exist" containerID="a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9" Oct 30 01:56:52.798167 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.798047 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9"} err="failed to get container status \"a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9\": rpc error: code = NotFound desc = could not find container \"a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9\": container with ID starting with a6be7d270aa32000effa84f7f95a09cce2c8fbc4eee2ec0829f0dfa89ee127e9 not found: ID does not exist" Oct 30 01:56:52.834826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.834794 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4"] Oct 30 01:56:52.845221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:52.845193 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-r9hg4"] Oct 30 01:56:53.752817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:53.752781 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" path="/var/lib/kubelet/pods/8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5/volumes" Oct 30 01:56:57.862736 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:57.862704 6275 generic.go:356] "Generic (PLEG): container finished" podID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerID="473665835c2b288b68e1b19f4889564031d9bed0b8b6a8c5e2350288926a046f" exitCode=0 Oct 30 01:56:57.863212 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:57.862781 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" event={"ID":"783ef77e-9b40-4eb0-8062-e35f4446011c","Type":"ContainerDied","Data":"473665835c2b288b68e1b19f4889564031d9bed0b8b6a8c5e2350288926a046f"} Oct 30 01:56:57.863280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:57.863262 6275 scope.go:122] "RemoveContainer" containerID="473665835c2b288b68e1b19f4889564031d9bed0b8b6a8c5e2350288926a046f" Oct 30 01:56:57.864950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:57.864918 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:57.865024 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:57.864990 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:57.865076 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:57.865023 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:56:58.984145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:58.981282 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" event={"ID":"783ef77e-9b40-4eb0-8062-e35f4446011c","Type":"ContainerStarted","Data":"e47d392c51964d92dcc1a03d63a59999559af40f52ac9950eb3d121abadb37ad"} Oct 30 01:56:59.697099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:59.697048 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:56:59.698218 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:59.698186 6275 patch_prober.go:29] interesting pod/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 01:56:59.698597 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:59.698570 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 01:56:59.699105 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:56:59.698512 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:57:00.696537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:00.696497 6275 patch_prober.go:29] interesting pod/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 01:57:00.697017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:00.696563 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 01:57:01.697753 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:01.697719 6275 patch_prober.go:29] interesting pod/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 01:57:01.698326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:01.698294 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 01:57:02.696912 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:02.696876 6275 patch_prober.go:29] interesting pod/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 01:57:02.697148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:02.696930 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 01:57:03.701410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:03.701371 6275 patch_prober.go:29] interesting pod/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 01:57:03.724538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:03.724491 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 01:57:04.393329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.389431 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g"] Oct 30 01:57:04.393329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.390565 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerName="payload-processing" Oct 30 01:57:04.393329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.390585 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerName="payload-processing" Oct 30 01:57:04.393329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.390797 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5b0c32-fc7c-4c32-bb0b-605b199ac4f5" containerName="payload-processing" Oct 30 01:57:04.402153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.396899 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g"] Oct 30 01:57:04.402153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.397009 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.424416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.424387 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2818a6\"" Oct 30 01:57:04.425143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.424855 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2818a6-dockercfg-jd6dl\"" Oct 30 01:57:04.471499 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.471462 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj47q\" (UniqueName: \"kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q\") pod \"payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.471687 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.471607 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.573575 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.573529 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj47q\" (UniqueName: \"kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q\") pod \"payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.574875 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.573643 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.575458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.575428 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.619277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.619247 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj47q\" (UniqueName: \"kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q\") pod \"payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:04.697443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.697352 6275 patch_prober.go:29] interesting pod/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 01:57:04.697718 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.697665 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 01:57:04.723132 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:04.723097 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:05.047538 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.047507 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g"] Oct 30 01:57:05.049372 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:05.049337 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356d78a7_e84d_4e78_b7bf_b14406f2243b.slice/crio-b5fd7a0f0ea409a6598e359e47ce465ed19490e35f180112d47e42a1975a9d0e WatchSource:0}: Error finding container b5fd7a0f0ea409a6598e359e47ce465ed19490e35f180112d47e42a1975a9d0e: Status 404 returned error can't find the container with id b5fd7a0f0ea409a6598e359e47ce465ed19490e35f180112d47e42a1975a9d0e Oct 30 01:57:05.093213 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.093182 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8"] Oct 30 01:57:05.130486 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.130455 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld"] Oct 30 01:57:05.420220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.419522 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" event={"ID":"356d78a7-e84d-4e78-b7bf-b14406f2243b","Type":"ContainerStarted","Data":"6e97a0eb9791054b21649a082fee50ad60bc592fada7ea1de2e10d58940cd633"} Oct 30 01:57:05.420220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.419595 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" event={"ID":"356d78a7-e84d-4e78-b7bf-b14406f2243b","Type":"ContainerStarted","Data":"b5fd7a0f0ea409a6598e359e47ce465ed19490e35f180112d47e42a1975a9d0e"} Oct 30 01:57:05.423485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.420882 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:05.700779 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.700704 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:57:05.701838 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.701817 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:57:05.747557 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:05.747509 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" podStartSLOduration=2.747496265 podStartE2EDuration="2.747496265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:05.471491552 +0000 UTC m=+1653.863299185" watchObservedRunningTime="2026-10-30 01:57:05.747496265 +0000 UTC m=+1654.139303898" Oct 30 01:57:10.093926 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.093882 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" podUID="09db4327-de14-4608-8793-ac3a294aec70" containerName="payload-processing" containerID="cri-o://ceb5ffedaff423be0a5c4c009697f7eeafe8fd6b19072c410a79d7ef1d7ca0e9" gracePeriod=25 Oct 30 01:57:10.130783 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.130730 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" podUID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" containerName="payload-pre-processing" containerID="cri-o://387ac8259702d77363e6db38c2452928b6bc746566956ecb048c47d4eadce0bb" gracePeriod=25 Oct 30 01:57:10.530817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.527592 6275 generic.go:356] "Generic (PLEG): container finished" podID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" containerID="387ac8259702d77363e6db38c2452928b6bc746566956ecb048c47d4eadce0bb" exitCode=0 Oct 30 01:57:10.530817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.527691 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" event={"ID":"5f22161c-ec4c-46bd-9503-fe3eabb96b6f","Type":"ContainerDied","Data":"387ac8259702d77363e6db38c2452928b6bc746566956ecb048c47d4eadce0bb"} Oct 30 01:57:10.530817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.529640 6275 generic.go:356] "Generic (PLEG): container finished" podID="09db4327-de14-4608-8793-ac3a294aec70" containerID="ceb5ffedaff423be0a5c4c009697f7eeafe8fd6b19072c410a79d7ef1d7ca0e9" exitCode=0 Oct 30 01:57:10.530817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.529716 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" event={"ID":"09db4327-de14-4608-8793-ac3a294aec70","Type":"ContainerDied","Data":"ceb5ffedaff423be0a5c4c009697f7eeafe8fd6b19072c410a79d7ef1d7ca0e9"} Oct 30 01:57:10.574503 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.574475 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:57:10.645723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.645664 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w\" (UniqueName: \"kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w\") pod \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " Oct 30 01:57:10.645884 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.645857 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume\") pod \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\" (UID: \"5f22161c-ec4c-46bd-9503-fe3eabb96b6f\") " Oct 30 01:57:10.646418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.646390 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume" pod "5f22161c-ec4c-46bd-9503-fe3eabb96b6f" (UID: "5f22161c-ec4c-46bd-9503-fe3eabb96b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:10.646856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.646833 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:10.648149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.648105 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:57:10.648379 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.648349 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w" pod "5f22161c-ec4c-46bd-9503-fe3eabb96b6f" (UID: "5f22161c-ec4c-46bd-9503-fe3eabb96b6f"). InnerVolumeSpecName "kube-api-access-bxp6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:10.747955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.747915 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume\") pod \"09db4327-de14-4608-8793-ac3a294aec70\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " Oct 30 01:57:10.748309 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.748286 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj\" (UniqueName: \"kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj\") pod \"09db4327-de14-4608-8793-ac3a294aec70\" (UID: \"09db4327-de14-4608-8793-ac3a294aec70\") " Oct 30 01:57:10.748683 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.748653 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume" pod "09db4327-de14-4608-8793-ac3a294aec70" (UID: "09db4327-de14-4608-8793-ac3a294aec70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:10.749084 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.749061 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxp6w\" (UniqueName: \"kubernetes.io/projected/5f22161c-ec4c-46bd-9503-fe3eabb96b6f-kube-api-access-bxp6w\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:10.749084 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.749085 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09db4327-de14-4608-8793-ac3a294aec70-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:10.751066 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.751037 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj" pod "09db4327-de14-4608-8793-ac3a294aec70" (UID: "09db4327-de14-4608-8793-ac3a294aec70"). InnerVolumeSpecName "kube-api-access-znkzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:10.850258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:10.850163 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znkzj\" (UniqueName: \"kubernetes.io/projected/09db4327-de14-4608-8793-ac3a294aec70-kube-api-access-znkzj\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:11.516540 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.516496 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" podUID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.94:50052\" within 1s: context deadline exceeded" Oct 30 01:57:11.516937 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:11.516622 6275 logging.go:55] [core] [Channel #885 SubChannel #886]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.94:50052", ServerName: "10.134.0.94:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.94:50052: operation was canceled" Oct 30 01:57:11.523401 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.522649 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" podUID="09db4327-de14-4608-8793-ac3a294aec70" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.95:50052\" within 1s: context deadline exceeded" Oct 30 01:57:11.523401 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:11.522825 6275 logging.go:55] [core] [Channel #887 SubChannel #888]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.95:50052", ServerName: "10.134.0.95:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.95:50052: operation was canceled" Oct 30 01:57:11.568680 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.568644 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" event={"ID":"5f22161c-ec4c-46bd-9503-fe3eabb96b6f","Type":"ContainerDied","Data":"fb614198a8c30ac86bb8a3b610c1c0711d779eff61f2cfd277c7a197ba3fa4f0"} Oct 30 01:57:11.568951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.568932 6275 scope.go:122] "RemoveContainer" containerID="387ac8259702d77363e6db38c2452928b6bc746566956ecb048c47d4eadce0bb" Oct 30 01:57:11.569245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.569225 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld" Oct 30 01:57:11.581614 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.581569 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" event={"ID":"09db4327-de14-4608-8793-ac3a294aec70","Type":"ContainerDied","Data":"b1c2701c2fb01b96d7a9f9e70145ab4590239bc7da860381226985dad1b4fb4f"} Oct 30 01:57:11.581938 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.581898 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8" Oct 30 01:57:11.627078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.626959 6275 scope.go:122] "RemoveContainer" containerID="ceb5ffedaff423be0a5c4c009697f7eeafe8fd6b19072c410a79d7ef1d7ca0e9" Oct 30 01:57:11.871715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.871633 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld"] Oct 30 01:57:11.876142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.874221 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922b2c-5f6bb77b59-mssld"] Oct 30 01:57:11.880494 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.880470 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8"] Oct 30 01:57:11.884848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:11.884797 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922b2c-66bf787d66-msht8"] Oct 30 01:57:13.767254 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:13.767218 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09db4327-de14-4608-8793-ac3a294aec70" path="/var/lib/kubelet/pods/09db4327-de14-4608-8793-ac3a294aec70/volumes" Oct 30 01:57:13.767963 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:13.767935 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" path="/var/lib/kubelet/pods/5f22161c-ec4c-46bd-9503-fe3eabb96b6f/volumes" Oct 30 01:57:15.009817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.009780 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd"] Oct 30 01:57:15.010587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.010566 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" containerName="payload-pre-processing" Oct 30 01:57:15.010769 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.010746 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" containerName="payload-pre-processing" Oct 30 01:57:15.010848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.010798 6275 cpu_manager.go:414] "containerMap: removing container" podUID="09db4327-de14-4608-8793-ac3a294aec70" containerName="payload-processing" Oct 30 01:57:15.010848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.010808 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09db4327-de14-4608-8793-ac3a294aec70" containerName="payload-processing" Oct 30 01:57:15.011047 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.011033 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f22161c-ec4c-46bd-9503-fe3eabb96b6f" containerName="payload-pre-processing" Oct 30 01:57:15.011085 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.011054 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="09db4327-de14-4608-8793-ac3a294aec70" containerName="payload-processing" Oct 30 01:57:15.016325 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.016306 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.033406 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.033385 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd098e\"" Oct 30 01:57:15.069175 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.069143 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd"] Oct 30 01:57:15.080234 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.080203 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb"] Oct 30 01:57:15.085898 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.085873 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.103380 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.103351 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb"] Oct 30 01:57:15.113908 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.113867 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd098e-dockercfg-fskhm\"" Oct 30 01:57:15.233238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.233193 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqwf\" (UniqueName: \"kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf\") pod \"payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.233534 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.233493 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.233806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.233785 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2ss\" (UniqueName: \"kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.234109 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.234093 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.400650 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.400494 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.402803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.376353 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.402803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.401104 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqwf\" (UniqueName: \"kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf\") pod \"payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.402803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.401153 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.402803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.401273 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2ss\" (UniqueName: \"kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.402803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.402735 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.513293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.513260 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2ss\" (UniqueName: \"kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:15.543656 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.542318 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqwf\" (UniqueName: \"kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf\") pod \"payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.631039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.631007 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:15.706189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:15.706157 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:16.089015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.088981 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd"] Oct 30 01:57:16.092210 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:16.091521 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbd0264_e6c8_43fd_8f79_6e2cf7afd1ba.slice/crio-22f8a07dfa021de392528fb4226adc30742ced3d6beacc6e517b63243e21be1d WatchSource:0}: Error finding container 22f8a07dfa021de392528fb4226adc30742ced3d6beacc6e517b63243e21be1d: Status 404 returned error can't find the container with id 22f8a07dfa021de392528fb4226adc30742ced3d6beacc6e517b63243e21be1d Oct 30 01:57:16.110833 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.110063 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb"] Oct 30 01:57:16.702209 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.702169 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" event={"ID":"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba","Type":"ContainerStarted","Data":"85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a"} Oct 30 01:57:16.702402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.702228 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" event={"ID":"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba","Type":"ContainerStarted","Data":"22f8a07dfa021de392528fb4226adc30742ced3d6beacc6e517b63243e21be1d"} Oct 30 01:57:16.702402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.702244 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" event={"ID":"ccd95087-68b2-4a95-8bb2-a3b66068f8f5","Type":"ContainerStarted","Data":"a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1"} Oct 30 01:57:16.702402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.702264 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:16.702402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.702299 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:16.702402 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.702311 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" event={"ID":"ccd95087-68b2-4a95-8bb2-a3b66068f8f5","Type":"ContainerStarted","Data":"672ffd158fc2ec5b864dffbea01e777843a65039faddb7343dcdd191f6a9a376"} Oct 30 01:57:16.709264 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:57:16.709229 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:57:16.791601 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.791540 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" podStartSLOduration=2.791522976 podStartE2EDuration="2.791522976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:16.748664223 +0000 UTC m=+1665.140471858" watchObservedRunningTime="2026-10-30 01:57:16.791522976 +0000 UTC m=+1665.183330611" Oct 30 01:57:16.792226 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:16.792177 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" podStartSLOduration=1.792165147 podStartE2EDuration="1.792165147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:16.789774753 +0000 UTC m=+1665.181582386" watchObservedRunningTime="2026-10-30 01:57:16.792165147 +0000 UTC m=+1665.183972781" Oct 30 01:57:17.436008 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.435980 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:57:17.593300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.593268 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b"] Oct 30 01:57:17.597820 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.597799 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.629899 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.628585 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2933ff00-serving-cert\"" Oct 30 01:57:17.673674 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.673506 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b"] Oct 30 01:57:17.695935 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.695860 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.696203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.696186 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.696329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.696317 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7s6q\" (UniqueName: \"kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.696567 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.696555 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.798271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.798218 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.798897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.798869 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.799077 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.799062 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.799222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.799207 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7s6q\" (UniqueName: \"kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.799818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.799796 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.801818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.801689 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.801818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.801761 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.823151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.823102 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7s6q\" (UniqueName: \"kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-svn7b\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:17.926150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:17.925820 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:18.385701 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:18.385672 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b"] Oct 30 01:57:18.387901 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:18.387863 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb7e2ad_a8c8_4765_b623_c97492224810.slice/crio-326ebc1f142dd566b08b30b04569191109460bbb4e83441a7167578f78f6bc23 WatchSource:0}: Error finding container 326ebc1f142dd566b08b30b04569191109460bbb4e83441a7167578f78f6bc23: Status 404 returned error can't find the container with id 326ebc1f142dd566b08b30b04569191109460bbb4e83441a7167578f78f6bc23 Oct 30 01:57:18.716577 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:18.716393 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" event={"ID":"fcb7e2ad-a8c8-4765-b623-c97492224810","Type":"ContainerStarted","Data":"326ebc1f142dd566b08b30b04569191109460bbb4e83441a7167578f78f6bc23"} Oct 30 01:57:19.760785 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:19.760743 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" event={"ID":"fcb7e2ad-a8c8-4765-b623-c97492224810","Type":"ContainerStarted","Data":"c5dbaf7e00527a57140d66ea1bdcfe910741c8b64db0d6cc48275de786aea852"} Oct 30 01:57:19.761288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:19.760811 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:19.803072 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:19.803022 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" podStartSLOduration=2.35753745 podStartE2EDuration="2.803009231s" totalImagesPullingTime="445.471781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:19.799169766 +0000 UTC m=+1668.190977400" watchObservedRunningTime="2026-10-30 01:57:19.803009231 +0000 UTC m=+1668.194816864" Oct 30 01:57:21.865286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:21.865243 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:57:21.865758 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:21.865301 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:57:23.786911 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:23.786875 6275 generic.go:356] "Generic (PLEG): container finished" podID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerID="320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5" exitCode=0 Oct 30 01:57:23.787397 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:23.786957 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" event={"ID":"9176efeb-3e9e-468b-bd0f-e5ee725a52f7","Type":"ContainerDied","Data":"320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5"} Oct 30 01:57:23.787553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:23.787532 6275 scope.go:122] "RemoveContainer" containerID="320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5" Oct 30 01:57:23.789470 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:23.789438 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:57:23.789600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:23.789514 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:57:23.789600 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:23.789555 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:57:24.370979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:24.370944 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p"] Oct 30 01:57:24.371321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:24.371197 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" podUID="01ffaabd-4733-4bfc-b7c7-03ae84aba138" containerName="maas-api" containerID="cri-o://51acc65621b498f63db71712dfe2fd09836604b6c4b487a399478a4d39e51175" gracePeriod=30 Oct 30 01:57:25.071956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.071920 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" event={"ID":"9176efeb-3e9e-468b-bd0f-e5ee725a52f7","Type":"ContainerStarted","Data":"23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369"} Oct 30 01:57:25.153239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.151179 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:57:25.215146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.209899 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:57:25.215146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.209962 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 01:57:25.261154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.237003 6275 generic.go:356] "Generic (PLEG): container finished" podID="01ffaabd-4733-4bfc-b7c7-03ae84aba138" containerID="51acc65621b498f63db71712dfe2fd09836604b6c4b487a399478a4d39e51175" exitCode=0 Oct 30 01:57:25.293595 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.237440 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" event={"ID":"01ffaabd-4733-4bfc-b7c7-03ae84aba138","Type":"ContainerDied","Data":"51acc65621b498f63db71712dfe2fd09836604b6c4b487a399478a4d39e51175"} Oct 30 01:57:25.293956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.293925 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" event={"ID":"01ffaabd-4733-4bfc-b7c7-03ae84aba138","Type":"ContainerDied","Data":"2bee9254b72ee4eb49ad173c04958945df152fad2e55c24c408fb0359006756f"} Oct 30 01:57:25.294186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.294167 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bee9254b72ee4eb49ad173c04958945df152fad2e55c24c408fb0359006756f" Oct 30 01:57:25.298416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.298394 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:57:25.406787 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.406694 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\") pod \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " Oct 30 01:57:25.406971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.406818 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t\" (UniqueName: \"kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t\") pod \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " Oct 30 01:57:25.406971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.406869 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca\") pod \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " Oct 30 01:57:25.406971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.406889 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs\") pod \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\" (UID: \"01ffaabd-4733-4bfc-b7c7-03ae84aba138\") " Oct 30 01:57:25.407979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.407951 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca" pod "01ffaabd-4733-4bfc-b7c7-03ae84aba138" (UID: "01ffaabd-4733-4bfc-b7c7-03ae84aba138"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:25.410518 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.410496 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t" pod "01ffaabd-4733-4bfc-b7c7-03ae84aba138" (UID: "01ffaabd-4733-4bfc-b7c7-03ae84aba138"). InnerVolumeSpecName "kube-api-access-wm44t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:25.410637 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.410500 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs" pod "01ffaabd-4733-4bfc-b7c7-03ae84aba138" (UID: "01ffaabd-4733-4bfc-b7c7-03ae84aba138"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:57:25.410695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.410517 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls" pod "01ffaabd-4733-4bfc-b7c7-03ae84aba138" (UID: "01ffaabd-4733-4bfc-b7c7-03ae84aba138"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:57:25.509165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.509112 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm44t\" (UniqueName: \"kubernetes.io/projected/01ffaabd-4733-4bfc-b7c7-03ae84aba138-kube-api-access-wm44t\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:25.509165 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.509166 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffaabd-4733-4bfc-b7c7-03ae84aba138-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:25.509377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.509181 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:25.509377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.509190 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffaabd-4733-4bfc-b7c7-03ae84aba138-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:25.823807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:25.823776 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:57:26.163893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:26.163782 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:57:26.163893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:26.163854 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 01:57:26.300957 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:26.300923 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p" Oct 30 01:57:26.368152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:26.368094 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p"] Oct 30 01:57:26.371499 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:26.371471 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922b2c-89c4f568b-kqn5p"] Oct 30 01:57:27.163144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:27.162698 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:57:27.163144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:27.162768 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 01:57:27.711391 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:27.711366 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:57:27.770898 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:27.770866 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ffaabd-4733-4bfc-b7c7-03ae84aba138" path="/var/lib/kubelet/pods/01ffaabd-4733-4bfc-b7c7-03ae84aba138/volumes" Oct 30 01:57:28.153531 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:28.153492 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:57:28.153913 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:28.153878 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 01:57:28.719829 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:28.719800 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:57:29.155186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:29.153555 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:57:29.155186 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:29.153635 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 01:57:30.184973 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:30.184939 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:57:30.186092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:30.186059 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:57:30.188787 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:30.188640 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:57:32.297805 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.289758 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr"] Oct 30 01:57:32.298573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.298524 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" containerID="cri-o://96af87fb7f78b6935beb17d479823163ddf84ee5c2d3a7120831bab0740e3896" gracePeriod=30 Oct 30 01:57:32.314297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.314261 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k"] Oct 30 01:57:32.314594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.314564 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" containerID="cri-o://4a7dd2109f129b573030338abcbacd0301b695b73e70e255058b734eacd6616b" gracePeriod=30 Oct 30 01:57:32.797668 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.797635 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng"] Oct 30 01:57:32.798456 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.798434 6275 cpu_manager.go:414] "containerMap: removing container" podUID="01ffaabd-4733-4bfc-b7c7-03ae84aba138" containerName="maas-api" Oct 30 01:57:32.798586 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.798457 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ffaabd-4733-4bfc-b7c7-03ae84aba138" containerName="maas-api" Oct 30 01:57:32.798645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.798610 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ffaabd-4733-4bfc-b7c7-03ae84aba138" containerName="maas-api" Oct 30 01:57:32.802464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.802439 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:32.816321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.816296 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bd098e-serving-cert\"" Oct 30 01:57:32.823640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.823615 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng"] Oct 30 01:57:32.928601 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.928564 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:32.928766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.928744 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:32.928815 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.928782 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bkl\" (UniqueName: \"kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:32.928874 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:32.928860 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.034271 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.034225 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.034770 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.034737 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.035167 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.035115 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.035366 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.035338 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bkl\" (UniqueName: \"kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.037099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.037080 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.070955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.070872 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bkl\" (UniqueName: \"kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.081781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.077089 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.081781 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.077604 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd098e-944fff597-dmbng\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.119802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.116863 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:33.438238 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.438213 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng"] Oct 30 01:57:33.440021 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:33.439995 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56adbf72_6b8c_4e60_bb05_be3cef4f4b3f.slice/crio-831a3977e045a5e6620ace343098b91065de649262d7935f6d602ced559205a1 WatchSource:0}: Error finding container 831a3977e045a5e6620ace343098b91065de649262d7935f6d602ced559205a1: Status 404 returned error can't find the container with id 831a3977e045a5e6620ace343098b91065de649262d7935f6d602ced559205a1 Oct 30 01:57:33.449635 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:33.449602 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" event={"ID":"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f","Type":"ContainerStarted","Data":"831a3977e045a5e6620ace343098b91065de649262d7935f6d602ced559205a1"} Oct 30 01:57:34.460174 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:34.458876 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" event={"ID":"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f","Type":"ContainerStarted","Data":"bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564"} Oct 30 01:57:34.460766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:34.460457 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:34.503473 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:34.503420 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" podStartSLOduration=2.168243997 podStartE2EDuration="2.503406783s" totalImagesPullingTime="335.162786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:34.49956359 +0000 UTC m=+1682.891371223" watchObservedRunningTime="2026-10-30 01:57:34.503406783 +0000 UTC m=+1682.895214440" Oct 30 01:57:37.592207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.591897 6275 generic.go:356] "Generic (PLEG): container finished" podID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerID="4a7dd2109f129b573030338abcbacd0301b695b73e70e255058b734eacd6616b" exitCode=0 Oct 30 01:57:37.592207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.592054 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" event={"ID":"4c718437-d257-46ba-a3d4-14f6aeb5d40e","Type":"ContainerDied","Data":"4a7dd2109f129b573030338abcbacd0301b695b73e70e255058b734eacd6616b"} Oct 30 01:57:37.595322 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.595294 6275 generic.go:356] "Generic (PLEG): container finished" podID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerID="96af87fb7f78b6935beb17d479823163ddf84ee5c2d3a7120831bab0740e3896" exitCode=0 Oct 30 01:57:37.595549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.595517 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" event={"ID":"62c71dd7-6c92-4f06-9501-7247cf7673a1","Type":"ContainerDied","Data":"96af87fb7f78b6935beb17d479823163ddf84ee5c2d3a7120831bab0740e3896"} Oct 30 01:57:37.653416 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.653377 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:57:37.699905 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.699859 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:57:37.805965 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.805918 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806008 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806110 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806202 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806239 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806282 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806287 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.806357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806327 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806378 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806420 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806560 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806524 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806652 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806697 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806824 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806728 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806824 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806762 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806824 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806792 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806838 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs\") pod \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\" (UID: \"4c718437-d257-46ba-a3d4-14f6aeb5d40e\") " Oct 30 01:57:37.806991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806843 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.806991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806870 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.806991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.806904 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token\") pod \"62c71dd7-6c92-4f06-9501-7247cf7673a1\" (UID: \"62c71dd7-6c92-4f06-9501-7247cf7673a1\") " Oct 30 01:57:37.807220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.807158 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.807528 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.807939 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.807964 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.807978 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.807992 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.809637 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.809743 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.809954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.809865 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.810493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.810036 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.810493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.810210 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:37.810691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.810659 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.811311 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.811277 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "kube-api-access-vxrjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.811948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.811921 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:37.812310 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.812288 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:37.812501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.812351 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.812501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.812494 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.812638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.812583 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy" pod "62c71dd7-6c92-4f06-9501-7247cf7673a1" (UID: "62c71dd7-6c92-4f06-9501-7247cf7673a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.812693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.812658 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "kube-api-access-8zdmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.814188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.814160 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo" pod "4c718437-d257-46ba-a3d4-14f6aeb5d40e" (UID: "4c718437-d257-46ba-a3d4-14f6aeb5d40e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:37.910050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.909965 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.909997 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910009 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910020 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910032 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910050 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910043 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c71dd7-6c92-4f06-9501-7247cf7673a1-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910056 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910070 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c718437-d257-46ba-a3d4-14f6aeb5d40e-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910082 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrjb\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-kube-api-access-vxrjb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910095 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdmf\" (UniqueName: \"kubernetes.io/projected/4c718437-d257-46ba-a3d4-14f6aeb5d40e-kube-api-access-8zdmf\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910106 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910116 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c718437-d257-46ba-a3d4-14f6aeb5d40e-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910167 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.910410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:37.910181 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c71dd7-6c92-4f06-9501-7247cf7673a1-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:38.685198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.676134 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" event={"ID":"4c718437-d257-46ba-a3d4-14f6aeb5d40e","Type":"ContainerDied","Data":"0d1e558cd79d659a872161b3de3c8c7495d3480b56bf52fa9b7e1afbf6d64291"} Oct 30 01:57:38.685198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.676226 6275 scope.go:122] "RemoveContainer" containerID="4a7dd2109f129b573030338abcbacd0301b695b73e70e255058b734eacd6616b" Oct 30 01:57:38.685198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.676456 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k" Oct 30 01:57:38.748427 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.721815 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" event={"ID":"62c71dd7-6c92-4f06-9501-7247cf7673a1","Type":"ContainerDied","Data":"e92bb5f887a9d0a01e1c660183686dd6056e52d7e1cf1081abee6e6da0bda980"} Oct 30 01:57:38.748427 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.721954 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr" Oct 30 01:57:38.832151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.831377 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k"] Oct 30 01:57:38.838499 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.838432 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-bhc9k"] Oct 30 01:57:38.896812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.896509 6275 scope.go:122] "RemoveContainer" containerID="96af87fb7f78b6935beb17d479823163ddf84ee5c2d3a7120831bab0740e3896" Oct 30 01:57:38.953871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.932375 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf"] Oct 30 01:57:38.953871 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.952750 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr"] Oct 30 01:57:38.958221 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:38.958192 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-jftcr"] Oct 30 01:57:39.799464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:39.799384 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" path="/var/lib/kubelet/pods/4c718437-d257-46ba-a3d4-14f6aeb5d40e/volumes" Oct 30 01:57:39.800567 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:39.800546 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" path="/var/lib/kubelet/pods/62c71dd7-6c92-4f06-9501-7247cf7673a1/volumes" Oct 30 01:57:41.615968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:41.615937 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 01:57:43.938151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:43.934324 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" podUID="c5aab290-ff40-45d2-9592-3d9b0f5fd56f" containerName="payload-pre-processing" containerID="cri-o://1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede" gracePeriod=25 Oct 30 01:57:44.345059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.345033 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:57:44.364369 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.364338 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume\") pod \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " Oct 30 01:57:44.364555 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.364513 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9\" (UniqueName: \"kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9\") pod \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\" (UID: \"c5aab290-ff40-45d2-9592-3d9b0f5fd56f\") " Oct 30 01:57:44.364795 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.364754 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume" pod "c5aab290-ff40-45d2-9592-3d9b0f5fd56f" (UID: "c5aab290-ff40-45d2-9592-3d9b0f5fd56f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:44.365761 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.365098 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:44.398061 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.394789 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9" pod "c5aab290-ff40-45d2-9592-3d9b0f5fd56f" (UID: "c5aab290-ff40-45d2-9592-3d9b0f5fd56f"). InnerVolumeSpecName "kube-api-access-kzdb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:44.469159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.466419 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdb9\" (UniqueName: \"kubernetes.io/projected/c5aab290-ff40-45d2-9592-3d9b0f5fd56f-kube-api-access-kzdb9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:44.870647 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.870614 6275 generic.go:356] "Generic (PLEG): container finished" podID="c5aab290-ff40-45d2-9592-3d9b0f5fd56f" containerID="1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede" exitCode=0 Oct 30 01:57:44.870844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.870660 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" Oct 30 01:57:44.870844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.870668 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" event={"ID":"c5aab290-ff40-45d2-9592-3d9b0f5fd56f","Type":"ContainerDied","Data":"1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede"} Oct 30 01:57:44.870844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.870720 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf" event={"ID":"c5aab290-ff40-45d2-9592-3d9b0f5fd56f","Type":"ContainerDied","Data":"3f5a0adafdce9206f0983e2807b534d2c79dd78c5f209e177be92452d098ca3d"} Oct 30 01:57:44.870844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.870748 6275 scope.go:122] "RemoveContainer" containerID="1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede" Oct 30 01:57:44.881869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.881849 6275 scope.go:122] "RemoveContainer" containerID="1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede" Oct 30 01:57:44.882143 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:57:44.882107 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede\": container with ID starting with 1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede not found: ID does not exist" containerID="1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede" Oct 30 01:57:44.882207 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.882151 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede"} err="failed to get container status \"1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede\": rpc error: code = NotFound desc = could not find container \"1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede\": container with ID starting with 1e6bf0a8a178bcad5340112fa9e136bf0684e0291eac20e4ad6dc02588528ede not found: ID does not exist" Oct 30 01:57:44.917774 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.917745 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf"] Oct 30 01:57:44.928949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:44.928922 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c11f-6fc664b8c-j4vvf"] Oct 30 01:57:45.740348 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:45.740318 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5aab290-ff40-45d2-9592-3d9b0f5fd56f" path="/var/lib/kubelet/pods/c5aab290-ff40-45d2-9592-3d9b0f5fd56f/volumes" Oct 30 01:57:46.697108 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:57:46.697075 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:57:53.270767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:53.270725 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4"] Oct 30 01:57:53.304752 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:53.304717 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5"] Oct 30 01:57:53.956845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:53.956801 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d"] Oct 30 01:57:53.957111 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:53.957069 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" podUID="9f553128-bc31-47f7-b454-89624973bfa9" containerName="maas-api" containerID="cri-o://0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222" gracePeriod=30 Oct 30 01:57:54.520077 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.520045 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:57:54.587146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.582486 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\") pod \"9f553128-bc31-47f7-b454-89624973bfa9\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " Oct 30 01:57:54.587146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.582546 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn\" (UniqueName: \"kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn\") pod \"9f553128-bc31-47f7-b454-89624973bfa9\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " Oct 30 01:57:54.587146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.582782 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs\") pod \"9f553128-bc31-47f7-b454-89624973bfa9\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " Oct 30 01:57:54.587146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.583074 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca\") pod \"9f553128-bc31-47f7-b454-89624973bfa9\" (UID: \"9f553128-bc31-47f7-b454-89624973bfa9\") " Oct 30 01:57:54.594143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.591414 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca" pod "9f553128-bc31-47f7-b454-89624973bfa9" (UID: "9f553128-bc31-47f7-b454-89624973bfa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:54.618533 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.618470 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls" pod "9f553128-bc31-47f7-b454-89624973bfa9" (UID: "9f553128-bc31-47f7-b454-89624973bfa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:57:54.619297 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.619253 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs" pod "9f553128-bc31-47f7-b454-89624973bfa9" (UID: "9f553128-bc31-47f7-b454-89624973bfa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:57:54.627448 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.627405 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn" pod "9f553128-bc31-47f7-b454-89624973bfa9" (UID: "9f553128-bc31-47f7-b454-89624973bfa9"). InnerVolumeSpecName "kube-api-access-6p5tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:54.697259 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.690554 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:54.697580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.697504 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f553128-bc31-47f7-b454-89624973bfa9-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:54.697712 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.697698 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f553128-bc31-47f7-b454-89624973bfa9-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:54.697817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.697806 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5tn\" (UniqueName: \"kubernetes.io/projected/9f553128-bc31-47f7-b454-89624973bfa9-kube-api-access-6p5tn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:54.958625 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.958529 6275 generic.go:356] "Generic (PLEG): container finished" podID="9f553128-bc31-47f7-b454-89624973bfa9" containerID="0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222" exitCode=0 Oct 30 01:57:54.958625 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.958584 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" Oct 30 01:57:54.958826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.958643 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" event={"ID":"9f553128-bc31-47f7-b454-89624973bfa9","Type":"ContainerDied","Data":"0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222"} Oct 30 01:57:54.958826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.958708 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d" event={"ID":"9f553128-bc31-47f7-b454-89624973bfa9","Type":"ContainerDied","Data":"f1657dc8e4e16ba65ce63af5d9a526e6b8756e556b4f3ca7062e545be0ca6b25"} Oct 30 01:57:54.958826 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.958739 6275 scope.go:122] "RemoveContainer" containerID="0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222" Oct 30 01:57:54.971520 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.971498 6275 scope.go:122] "RemoveContainer" containerID="0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222" Oct 30 01:57:54.971830 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:57:54.971808 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222\": container with ID starting with 0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222 not found: ID does not exist" containerID="0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222" Oct 30 01:57:54.971934 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:54.971839 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222"} err="failed to get container status \"0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222\": rpc error: code = NotFound desc = could not find container \"0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222\": container with ID starting with 0976e6ade3841bee1d7a96756b88906fa9283292f24c8985a57ab453fef34222 not found: ID does not exist" Oct 30 01:57:55.007061 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:55.007030 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d"] Oct 30 01:57:55.010596 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:55.010568 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c11f-6ff64bbc98-mtt5d"] Oct 30 01:57:55.740999 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:55.740954 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f553128-bc31-47f7-b454-89624973bfa9" path="/var/lib/kubelet/pods/9f553128-bc31-47f7-b454-89624973bfa9/volumes" Oct 30 01:57:57.395929 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:57.395890 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv"] Oct 30 01:57:57.396437 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:57.396220 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" containerID="cri-o://23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369" gracePeriod=30 Oct 30 01:57:58.231183 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.231141 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd"] Oct 30 01:57:58.265056 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.265015 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb"] Oct 30 01:57:58.271180 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.271146 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" podUID="f6837615-dbb6-431c-9052-070cd52a5772" containerName="payload-processing" containerID="cri-o://fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518" gracePeriod=25 Oct 30 01:57:58.305616 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.305579 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" podUID="9b9c3dee-8334-40ed-a4b5-3c185a8573ae" containerName="payload-pre-processing" containerID="cri-o://d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69" gracePeriod=25 Oct 30 01:57:58.688007 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.687978 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:57:58.730734 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.730705 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:57:58.756504 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.756466 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg\" (UniqueName: \"kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg\") pod \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " Oct 30 01:57:58.756682 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.756577 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume\") pod \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\" (UID: \"9b9c3dee-8334-40ed-a4b5-3c185a8573ae\") " Oct 30 01:57:58.756682 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.756637 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume\") pod \"f6837615-dbb6-431c-9052-070cd52a5772\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " Oct 30 01:57:58.756812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.756705 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn\" (UniqueName: \"kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn\") pod \"f6837615-dbb6-431c-9052-070cd52a5772\" (UID: \"f6837615-dbb6-431c-9052-070cd52a5772\") " Oct 30 01:57:58.757159 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.757135 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume" pod "f6837615-dbb6-431c-9052-070cd52a5772" (UID: "f6837615-dbb6-431c-9052-070cd52a5772"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:58.757272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.757209 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume" pod "9b9c3dee-8334-40ed-a4b5-3c185a8573ae" (UID: "9b9c3dee-8334-40ed-a4b5-3c185a8573ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:58.757406 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.757385 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:58.757466 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.757411 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6837615-dbb6-431c-9052-070cd52a5772-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:58.759081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.759053 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn" pod "f6837615-dbb6-431c-9052-070cd52a5772" (UID: "f6837615-dbb6-431c-9052-070cd52a5772"). InnerVolumeSpecName "kube-api-access-5wrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:58.762036 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.762006 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg" pod "9b9c3dee-8334-40ed-a4b5-3c185a8573ae" (UID: "9b9c3dee-8334-40ed-a4b5-3c185a8573ae"). InnerVolumeSpecName "kube-api-access-l6zwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:58.820269 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.820178 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647"] Oct 30 01:57:58.821048 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821022 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9b9c3dee-8334-40ed-a4b5-3c185a8573ae" containerName="payload-pre-processing" Oct 30 01:57:58.821158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821049 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9c3dee-8334-40ed-a4b5-3c185a8573ae" containerName="payload-pre-processing" Oct 30 01:57:58.821158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821079 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c5aab290-ff40-45d2-9592-3d9b0f5fd56f" containerName="payload-pre-processing" Oct 30 01:57:58.821158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821085 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5aab290-ff40-45d2-9592-3d9b0f5fd56f" containerName="payload-pre-processing" Oct 30 01:57:58.821158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821105 6275 cpu_manager.go:414] "containerMap: removing container" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" Oct 30 01:57:58.821158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821109 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821182 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821190 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821205 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f6837615-dbb6-431c-9052-070cd52a5772" containerName="payload-processing" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821211 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6837615-dbb6-431c-9052-070cd52a5772" containerName="payload-processing" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821217 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9f553128-bc31-47f7-b454-89624973bfa9" containerName="maas-api" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821222 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f553128-bc31-47f7-b454-89624973bfa9" containerName="maas-api" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821330 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c718437-d257-46ba-a3d4-14f6aeb5d40e" containerName="istio-proxy" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821342 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6837615-dbb6-431c-9052-070cd52a5772" containerName="payload-processing" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821348 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f553128-bc31-47f7-b454-89624973bfa9" containerName="maas-api" Oct 30 01:57:58.821357 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821356 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9c3dee-8334-40ed-a4b5-3c185a8573ae" containerName="payload-pre-processing" Oct 30 01:57:58.821684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821364 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5aab290-ff40-45d2-9592-3d9b0f5fd56f" containerName="payload-pre-processing" Oct 30 01:57:58.821684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.821372 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c71dd7-6c92-4f06-9501-7247cf7673a1" containerName="istio-proxy" Oct 30 01:57:58.824236 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.824211 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.858649 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.858613 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.858916 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.858677 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859539 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859506 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859677 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859592 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859737 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859678 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859737 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859713 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjnvk\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859737 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859732 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859764 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.859897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859856 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.860000 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859972 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zwg\" (UniqueName: \"kubernetes.io/projected/9b9c3dee-8334-40ed-a4b5-3c185a8573ae-kube-api-access-l6zwg\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:58.860000 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.859987 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrjn\" (UniqueName: \"kubernetes.io/projected/f6837615-dbb6-431c-9052-070cd52a5772-kube-api-access-5wrjn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:57:58.860474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.860452 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647"] Oct 30 01:57:58.922018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.921990 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7542fa-openshift-default-dockercfg-4z4rq\"" Oct 30 01:57:58.960858 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.960823 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961019 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.960968 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961019 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961015 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961162 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961100 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnvk\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961258 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961320 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961444 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961451 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961536 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961550 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961599 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961839 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961849 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.961962 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.961888 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.963660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.963636 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.964066 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.964049 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.983046 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.983024 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.983227 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.983209 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnvk\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk\") pod \"e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:58.988869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.988842 6275 generic.go:356] "Generic (PLEG): container finished" podID="9b9c3dee-8334-40ed-a4b5-3c185a8573ae" containerID="d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69" exitCode=0 Oct 30 01:57:58.988979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.988892 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" Oct 30 01:57:58.988979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.988923 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" event={"ID":"9b9c3dee-8334-40ed-a4b5-3c185a8573ae","Type":"ContainerDied","Data":"d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69"} Oct 30 01:57:58.989067 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.988977 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5" event={"ID":"9b9c3dee-8334-40ed-a4b5-3c185a8573ae","Type":"ContainerDied","Data":"7a13bcbfe3825c8e99ba82f82236763d997bcc10d66f092a68449712bb4b92d0"} Oct 30 01:57:58.989067 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.988996 6275 scope.go:122] "RemoveContainer" containerID="d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69" Oct 30 01:57:58.990616 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.990539 6275 generic.go:356] "Generic (PLEG): container finished" podID="f6837615-dbb6-431c-9052-070cd52a5772" containerID="fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518" exitCode=0 Oct 30 01:57:58.990616 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.990602 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" Oct 30 01:57:58.990747 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.990603 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" event={"ID":"f6837615-dbb6-431c-9052-070cd52a5772","Type":"ContainerDied","Data":"fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518"} Oct 30 01:57:58.990747 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:58.990733 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4" event={"ID":"f6837615-dbb6-431c-9052-070cd52a5772","Type":"ContainerDied","Data":"3e48d8c6ac26e5b501ae96aef43f9413eec4db12eff00374027c8d6e7633b96a"} Oct 30 01:57:59.001522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.001482 6275 scope.go:122] "RemoveContainer" containerID="d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69" Oct 30 01:57:59.001826 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:57:59.001804 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69\": container with ID starting with d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69 not found: ID does not exist" containerID="d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69" Oct 30 01:57:59.001925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.001834 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69"} err="failed to get container status \"d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69\": rpc error: code = NotFound desc = could not find container \"d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69\": container with ID starting with d17649329d3e3d52cb0bb1a2f641f228f2f01ae06ac52c46eb1bd4337b7e7a69 not found: ID does not exist" Oct 30 01:57:59.001925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.001852 6275 scope.go:122] "RemoveContainer" containerID="fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518" Oct 30 01:57:59.033168 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.023408 6275 scope.go:122] "RemoveContainer" containerID="fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518" Oct 30 01:57:59.034342 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:57:59.034254 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518\": container with ID starting with fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518 not found: ID does not exist" containerID="fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518" Oct 30 01:57:59.034342 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.034297 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518"} err="failed to get container status \"fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518\": rpc error: code = NotFound desc = could not find container \"fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518\": container with ID starting with fa99c27abfd12fbaf69c2d61d2e0e0865ddeb70a64cc38e7f3fa65c2b6bf8518 not found: ID does not exist" Oct 30 01:57:59.096281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.096183 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5"] Oct 30 01:57:59.136515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.136479 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c93367-967fd786c-r7vg5"] Oct 30 01:57:59.155154 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.145580 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:57:59.210253 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.210180 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4"] Oct 30 01:57:59.315306 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.308775 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c93367-6666bff5bd-8pcq4"] Oct 30 01:57:59.740735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.740663 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9c3dee-8334-40ed-a4b5-3c185a8573ae" path="/var/lib/kubelet/pods/9b9c3dee-8334-40ed-a4b5-3c185a8573ae/volumes" Oct 30 01:57:59.741147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.741115 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6837615-dbb6-431c-9052-070cd52a5772" path="/var/lib/kubelet/pods/f6837615-dbb6-431c-9052-070cd52a5772/volumes" Oct 30 01:57:59.794017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.793989 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647"] Oct 30 01:57:59.796498 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:57:59.796472 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3afe93_cb6b_473e_85d5_4fae2ed27555.slice/crio-2330916e8e9c8c329629ca7bca9bb747b5730be3d30e5236b7f92310f17694ec WatchSource:0}: Error finding container 2330916e8e9c8c329629ca7bca9bb747b5730be3d30e5236b7f92310f17694ec: Status 404 returned error can't find the container with id 2330916e8e9c8c329629ca7bca9bb747b5730be3d30e5236b7f92310f17694ec Oct 30 01:57:59.799333 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.799291 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:57:59.799447 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.799364 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:57:59.799447 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:57:59.799396 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:00.000483 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.000388 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" event={"ID":"6d3afe93-cb6b-473e-85d5-4fae2ed27555","Type":"ContainerStarted","Data":"be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d"} Oct 30 01:58:00.000483 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.000452 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" event={"ID":"6d3afe93-cb6b-473e-85d5-4fae2ed27555","Type":"ContainerStarted","Data":"2330916e8e9c8c329629ca7bca9bb747b5730be3d30e5236b7f92310f17694ec"} Oct 30 01:58:00.052712 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.052648 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podStartSLOduration=2.052631198 podStartE2EDuration="2.052631198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:00.051152714 +0000 UTC m=+1708.442960338" watchObservedRunningTime="2026-10-30 01:58:00.052631198 +0000 UTC m=+1708.444438831" Oct 30 01:58:00.152294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.152255 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:00.153622 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.153589 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:00.153767 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.153638 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:00.204363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.204320 6275 patch_prober.go:29] interesting pod/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:58:00.204649 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:00.204612 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:58:01.156958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:01.156929 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:02.035462 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.035431 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:02.036315 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.036291 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:02.685795 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.685768 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:58:02.766861 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.766826 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.766951 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767068 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767025 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767107 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767206 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767165 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767312 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767230 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767312 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767298 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767402 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.767502 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.767472 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token\") pod \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\" (UID: \"9176efeb-3e9e-468b-bd0f-e5ee725a52f7\") " Oct 30 01:58:02.769010 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.768959 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:02.769200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.769139 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:02.769200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.769152 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:02.769336 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.769220 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:02.769485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.769464 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:02.786802 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.786722 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:02.803624 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.803297 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:02.813801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.813744 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:58:02.814418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.814384 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45" pod "9176efeb-3e9e-468b-bd0f-e5ee725a52f7" (UID: "9176efeb-3e9e-468b-bd0f-e5ee725a52f7"). InnerVolumeSpecName "kube-api-access-xjq45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870629 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870664 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870689 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870835 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870852 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870874 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjq45\" (UniqueName: \"kubernetes.io/projected/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-kube-api-access-xjq45\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870891 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870904 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:02.872330 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:02.870917 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9176efeb-3e9e-468b-bd0f-e5ee725a52f7-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:03.042724 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.042688 6275 generic.go:356] "Generic (PLEG): container finished" podID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerID="23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369" exitCode=0 Oct 30 01:58:03.042903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.042751 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" Oct 30 01:58:03.042903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.042787 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" event={"ID":"9176efeb-3e9e-468b-bd0f-e5ee725a52f7","Type":"ContainerDied","Data":"23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369"} Oct 30 01:58:03.042903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.042870 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv" event={"ID":"9176efeb-3e9e-468b-bd0f-e5ee725a52f7","Type":"ContainerDied","Data":"0d2c8762a6d87d88d706d5821c451a91618682442f56157fd0ef1678079a67d5"} Oct 30 01:58:03.042903 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.042894 6275 scope.go:122] "RemoveContainer" containerID="23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369" Oct 30 01:58:03.053660 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.053644 6275 scope.go:122] "RemoveContainer" containerID="320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5" Oct 30 01:58:03.063288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.063270 6275 scope.go:122] "RemoveContainer" containerID="23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369" Oct 30 01:58:03.063582 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:03.063557 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369\": container with ID starting with 23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369 not found: ID does not exist" containerID="23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369" Oct 30 01:58:03.063646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.063596 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369"} err="failed to get container status \"23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369\": rpc error: code = NotFound desc = could not find container \"23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369\": container with ID starting with 23c1e72a11df17d3fd26e82b28555568d576a6b30a2fd8a11f5c4a58f03b1369 not found: ID does not exist" Oct 30 01:58:03.063646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.063620 6275 scope.go:122] "RemoveContainer" containerID="320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5" Oct 30 01:58:03.063879 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:03.063855 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5\": container with ID starting with 320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5 not found: ID does not exist" containerID="320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5" Oct 30 01:58:03.063919 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.063885 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5"} err="failed to get container status \"320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5\": rpc error: code = NotFound desc = could not find container \"320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5\": container with ID starting with 320d54d3598e3e8a5e519cf7088420d51559df26512725f8851ddd546cf257c5 not found: ID does not exist" Oct 30 01:58:03.116474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.116439 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv"] Oct 30 01:58:03.126081 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.126050 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b2c11f-openshift-default-7c88c4b97c-jnwcv"] Oct 30 01:58:03.232373 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.232324 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" podUID="6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" containerName="payload-processing" containerID="cri-o://85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a" gracePeriod=25 Oct 30 01:58:03.266277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.266233 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" podUID="ccd95087-68b2-4a95-8bb2-a3b66068f8f5" containerName="payload-pre-processing" containerID="cri-o://a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1" gracePeriod=25 Oct 30 01:58:03.586011 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.585946 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:58:03.623406 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.623381 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:58:03.683032 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683001 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume\") pod \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " Oct 30 01:58:03.683230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683157 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss\" (UniqueName: \"kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss\") pod \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\" (UID: \"ccd95087-68b2-4a95-8bb2-a3b66068f8f5\") " Oct 30 01:58:03.683300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683259 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf\" (UniqueName: \"kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf\") pod \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " Oct 30 01:58:03.683361 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683349 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume\") pod \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\" (UID: \"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba\") " Oct 30 01:58:03.683501 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683471 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume" pod "ccd95087-68b2-4a95-8bb2-a3b66068f8f5" (UID: "ccd95087-68b2-4a95-8bb2-a3b66068f8f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:03.683945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683916 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume" pod "6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" (UID: "6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:03.684071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.683986 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:03.686036 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.686012 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss" pod "ccd95087-68b2-4a95-8bb2-a3b66068f8f5" (UID: "ccd95087-68b2-4a95-8bb2-a3b66068f8f5"). InnerVolumeSpecName "kube-api-access-xn2ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:03.686430 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.686082 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf" pod "6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" (UID: "6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba"). InnerVolumeSpecName "kube-api-access-kjqwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:03.740950 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.740912 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" path="/var/lib/kubelet/pods/9176efeb-3e9e-468b-bd0f-e5ee725a52f7/volumes" Oct 30 01:58:03.784991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.784952 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqwf\" (UniqueName: \"kubernetes.io/projected/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-kube-api-access-kjqwf\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:03.784991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.784983 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:03.784991 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:03.784994 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2ss\" (UniqueName: \"kubernetes.io/projected/ccd95087-68b2-4a95-8bb2-a3b66068f8f5-kube-api-access-xn2ss\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:04.055522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.055485 6275 generic.go:356] "Generic (PLEG): container finished" podID="ccd95087-68b2-4a95-8bb2-a3b66068f8f5" containerID="a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1" exitCode=0 Oct 30 01:58:04.055735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.055528 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" Oct 30 01:58:04.055735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.055564 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" event={"ID":"ccd95087-68b2-4a95-8bb2-a3b66068f8f5","Type":"ContainerDied","Data":"a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1"} Oct 30 01:58:04.055735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.055623 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb" event={"ID":"ccd95087-68b2-4a95-8bb2-a3b66068f8f5","Type":"ContainerDied","Data":"672ffd158fc2ec5b864dffbea01e777843a65039faddb7343dcdd191f6a9a376"} Oct 30 01:58:04.055735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.055654 6275 scope.go:122] "RemoveContainer" containerID="a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1" Oct 30 01:58:04.057552 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.057525 6275 generic.go:356] "Generic (PLEG): container finished" podID="6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" containerID="85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a" exitCode=0 Oct 30 01:58:04.057709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.057672 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" event={"ID":"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba","Type":"ContainerDied","Data":"85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a"} Oct 30 01:58:04.057817 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.057719 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" event={"ID":"6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba","Type":"ContainerDied","Data":"22f8a07dfa021de392528fb4226adc30742ced3d6beacc6e517b63243e21be1d"} Oct 30 01:58:04.058278 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.058258 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd" Oct 30 01:58:04.067449 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.067423 6275 scope.go:122] "RemoveContainer" containerID="a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1" Oct 30 01:58:04.067789 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:04.067768 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1\": container with ID starting with a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1 not found: ID does not exist" containerID="a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1" Oct 30 01:58:04.067867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.067799 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1"} err="failed to get container status \"a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1\": rpc error: code = NotFound desc = could not find container \"a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1\": container with ID starting with a9f580b45ff09f1f0b769ac16d324568fd3dfe10e6e423ac0a621cf958a576f1 not found: ID does not exist" Oct 30 01:58:04.067867 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.067817 6275 scope.go:122] "RemoveContainer" containerID="85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a" Oct 30 01:58:04.080604 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.080580 6275 scope.go:122] "RemoveContainer" containerID="85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a" Oct 30 01:58:04.080967 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:04.080934 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a\": container with ID starting with 85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a not found: ID does not exist" containerID="85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a" Oct 30 01:58:04.081051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.080975 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a"} err="failed to get container status \"85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a\": rpc error: code = NotFound desc = could not find container \"85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a\": container with ID starting with 85b4b3507f701929ed0facc1a3487975a6a944ca797462168f777c75fad17c4a not found: ID does not exist" Oct 30 01:58:04.105956 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.105922 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd"] Oct 30 01:58:04.115807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.115781 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-676766d5fb-lfkpd"] Oct 30 01:58:04.159039 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.159002 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb"] Oct 30 01:58:04.175379 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.175349 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-5dc78f5475-2drtb"] Oct 30 01:58:04.359855 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.359777 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n"] Oct 30 01:58:04.360456 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360440 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ccd95087-68b2-4a95-8bb2-a3b66068f8f5" containerName="payload-pre-processing" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360458 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd95087-68b2-4a95-8bb2-a3b66068f8f5" containerName="payload-pre-processing" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360471 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" containerName="payload-processing" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360476 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" containerName="payload-processing" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360486 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360491 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360511 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" Oct 30 01:58:04.360542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360516 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" Oct 30 01:58:04.360780 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360664 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd95087-68b2-4a95-8bb2-a3b66068f8f5" containerName="payload-pre-processing" Oct 30 01:58:04.360780 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360683 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" containerName="payload-processing" Oct 30 01:58:04.360780 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360694 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" Oct 30 01:58:04.360780 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.360705 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9176efeb-3e9e-468b-bd0f-e5ee725a52f7" containerName="istio-proxy" Oct 30 01:58:04.365996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.365974 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.381691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.381670 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7542fa-dockercfg-dljf8\"" Oct 30 01:58:04.381780 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.381767 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7542fa\"" Oct 30 01:58:04.384939 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.384918 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n"] Oct 30 01:58:04.520171 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.512915 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtfm\" (UniqueName: \"kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.520171 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.513067 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.539228 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.539179 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc"] Oct 30 01:58:04.654198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.634485 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtfm\" (UniqueName: \"kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.654198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.634614 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.654198 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.635923 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.673313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.673280 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc"] Oct 30 01:58:04.682248 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.674267 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.739146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.738916 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.739146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.739103 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzt66\" (UniqueName: \"kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66\") pod \"payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.800179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.798208 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtfm\" (UniqueName: \"kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:04.852161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.848874 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.852161 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.849101 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzt66\" (UniqueName: \"kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66\") pod \"payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.923897 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.923813 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.928643 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.928618 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzt66\" (UniqueName: \"kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66\") pod \"payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:04.980844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:04.980811 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:05.148082 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.148041 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:05.288175 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.288145 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n"] Oct 30 01:58:05.291163 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:05.290749 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb21048_65b3_4aad_8431_e1538619a298.slice/crio-b7c8d3dad3d209f9bdc864e393bee0fd213cf0855eb21fdd962e2ff9d6cad5f5 WatchSource:0}: Error finding container b7c8d3dad3d209f9bdc864e393bee0fd213cf0855eb21fdd962e2ff9d6cad5f5: Status 404 returned error can't find the container with id b7c8d3dad3d209f9bdc864e393bee0fd213cf0855eb21fdd962e2ff9d6cad5f5 Oct 30 01:58:05.459801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.459700 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc"] Oct 30 01:58:05.469523 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:05.469494 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbb3956_0e5c_40b9_b78f_77f1a307148e.slice/crio-e97d33c9c2509972328d012ae51b019c7872f58114aacb0952dc22a4422fac3c WatchSource:0}: Error finding container e97d33c9c2509972328d012ae51b019c7872f58114aacb0952dc22a4422fac3c: Status 404 returned error can't find the container with id e97d33c9c2509972328d012ae51b019c7872f58114aacb0952dc22a4422fac3c Oct 30 01:58:05.740544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.740448 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba" path="/var/lib/kubelet/pods/6dbd0264-e6c8-43fd-8f79-6e2cf7afd1ba/volumes" Oct 30 01:58:05.741112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.741091 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd95087-68b2-4a95-8bb2-a3b66068f8f5" path="/var/lib/kubelet/pods/ccd95087-68b2-4a95-8bb2-a3b66068f8f5/volumes" Oct 30 01:58:05.917373 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.917338 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km"] Oct 30 01:58:05.917571 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:05.917547 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" podUID="dc7fe934-149b-4b99-b5a7-d33262bfc32d" containerName="maas-api" containerID="cri-o://28b3582d8f3b1313ed190ae5be195bf8e33ef48aeff7e34862516fbf1484e764" gracePeriod=30 Oct 30 01:58:06.110984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.110947 6275 generic.go:356] "Generic (PLEG): container finished" podID="dc7fe934-149b-4b99-b5a7-d33262bfc32d" containerID="28b3582d8f3b1313ed190ae5be195bf8e33ef48aeff7e34862516fbf1484e764" exitCode=0 Oct 30 01:58:06.111187 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.111113 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" event={"ID":"dc7fe934-149b-4b99-b5a7-d33262bfc32d","Type":"ContainerDied","Data":"28b3582d8f3b1313ed190ae5be195bf8e33ef48aeff7e34862516fbf1484e764"} Oct 30 01:58:06.115458 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.115339 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" event={"ID":"9cbb3956-0e5c-40b9-b78f-77f1a307148e","Type":"ContainerStarted","Data":"2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743"} Oct 30 01:58:06.116746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.116710 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" event={"ID":"9cbb3956-0e5c-40b9-b78f-77f1a307148e","Type":"ContainerStarted","Data":"e97d33c9c2509972328d012ae51b019c7872f58114aacb0952dc22a4422fac3c"} Oct 30 01:58:06.116856 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.116775 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:06.117846 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.117794 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" event={"ID":"5eb21048-65b3-4aad-8431-e1538619a298","Type":"ContainerStarted","Data":"a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a"} Oct 30 01:58:06.117846 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.117853 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" event={"ID":"5eb21048-65b3-4aad-8431-e1538619a298","Type":"ContainerStarted","Data":"b7c8d3dad3d209f9bdc864e393bee0fd213cf0855eb21fdd962e2ff9d6cad5f5"} Oct 30 01:58:06.118601 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.118579 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:06.263146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.260408 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" podStartSLOduration=2.260386747 podStartE2EDuration="2.260386747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:06.22950088 +0000 UTC m=+1714.621308527" watchObservedRunningTime="2026-10-30 01:58:06.260386747 +0000 UTC m=+1714.652194381" Oct 30 01:58:06.306200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.292538 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" podStartSLOduration=2.292522403 podStartE2EDuration="2.292522403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:06.290435284 +0000 UTC m=+1714.682242921" watchObservedRunningTime="2026-10-30 01:58:06.292522403 +0000 UTC m=+1714.684330036" Oct 30 01:58:06.530149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.526941 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:58:06.619203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.616475 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb\" (UniqueName: \"kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb\") pod \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " Oct 30 01:58:06.619203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.616548 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls\") pod \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " Oct 30 01:58:06.619203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.616674 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs\") pod \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " Oct 30 01:58:06.619203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.616707 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca\") pod \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\" (UID: \"dc7fe934-149b-4b99-b5a7-d33262bfc32d\") " Oct 30 01:58:06.620681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.620647 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca" pod "dc7fe934-149b-4b99-b5a7-d33262bfc32d" (UID: "dc7fe934-149b-4b99-b5a7-d33262bfc32d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:06.629045 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.629014 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb" pod "dc7fe934-149b-4b99-b5a7-d33262bfc32d" (UID: "dc7fe934-149b-4b99-b5a7-d33262bfc32d"). InnerVolumeSpecName "kube-api-access-6hqbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:06.632418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.632370 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs" pod "dc7fe934-149b-4b99-b5a7-d33262bfc32d" (UID: "dc7fe934-149b-4b99-b5a7-d33262bfc32d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:06.648663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.648628 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls" pod "dc7fe934-149b-4b99-b5a7-d33262bfc32d" (UID: "dc7fe934-149b-4b99-b5a7-d33262bfc32d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:06.721645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.721545 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hqbb\" (UniqueName: \"kubernetes.io/projected/dc7fe934-149b-4b99-b5a7-d33262bfc32d-kube-api-access-6hqbb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:06.721645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.721585 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:06.721645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.721600 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fe934-149b-4b99-b5a7-d33262bfc32d-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:06.721645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:06.721612 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fe934-149b-4b99-b5a7-d33262bfc32d-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:07.126705 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:07.126672 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" Oct 30 01:58:07.127204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:07.126709 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km" event={"ID":"dc7fe934-149b-4b99-b5a7-d33262bfc32d","Type":"ContainerDied","Data":"626c113db066dae766ccf25490a792016c2c893152ce5e98c25d95c2155c278d"} Oct 30 01:58:07.127204 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:07.126786 6275 scope.go:122] "RemoveContainer" containerID="28b3582d8f3b1313ed190ae5be195bf8e33ef48aeff7e34862516fbf1484e764" Oct 30 01:58:07.184383 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:07.184349 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km"] Oct 30 01:58:07.194383 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:07.194355 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c93367-8794dc4b7-nk7km"] Oct 30 01:58:07.741700 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:07.741667 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7fe934-149b-4b99-b5a7-d33262bfc32d" path="/var/lib/kubelet/pods/dc7fe934-149b-4b99-b5a7-d33262bfc32d/volumes" Oct 30 01:58:11.875772 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:11.875678 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4"] Oct 30 01:58:11.876413 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:11.875972 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" containerID="cri-o://e47d392c51964d92dcc1a03d63a59999559af40f52ac9950eb3d121abadb37ad" gracePeriod=30 Oct 30 01:58:13.028499 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.028456 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c"] Oct 30 01:58:13.032723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.029486 6275 cpu_manager.go:414] "containerMap: removing container" podUID="dc7fe934-149b-4b99-b5a7-d33262bfc32d" containerName="maas-api" Oct 30 01:58:13.032723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.029512 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7fe934-149b-4b99-b5a7-d33262bfc32d" containerName="maas-api" Oct 30 01:58:13.032723 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.029689 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7fe934-149b-4b99-b5a7-d33262bfc32d" containerName="maas-api" Oct 30 01:58:13.036152 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.035766 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.071594 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.067844 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd098e-dockercfg-8skhf\"" Oct 30 01:58:13.078146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.075087 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c"] Oct 30 01:58:13.078146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.075276 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd098e\"" Oct 30 01:58:13.094185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.092836 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz92f\" (UniqueName: \"kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f\") pod \"payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.094185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.093015 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.194612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.194573 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz92f\" (UniqueName: \"kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f\") pod \"payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.194914 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.194890 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.195542 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.195518 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.217268 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.217240 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz92f\" (UniqueName: \"kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f\") pod \"payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.396879 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.396788 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:13.711667 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:13.711643 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c"] Oct 30 01:58:13.714250 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:13.714223 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c7bf49_436b_4bcf_b997_d5b3b49fbe8e.slice/crio-2e4ec5b917260babcb29756b05c0fc389bc853fe194ccc080331efbfb86d5414 WatchSource:0}: Error finding container 2e4ec5b917260babcb29756b05c0fc389bc853fe194ccc080331efbfb86d5414: Status 404 returned error can't find the container with id 2e4ec5b917260babcb29756b05c0fc389bc853fe194ccc080331efbfb86d5414 Oct 30 01:58:14.227418 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:14.227382 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" event={"ID":"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e","Type":"ContainerStarted","Data":"2e4ec5b917260babcb29756b05c0fc389bc853fe194ccc080331efbfb86d5414"} Oct 30 01:58:16.243495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:16.243458 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" event={"ID":"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e","Type":"ContainerStarted","Data":"1ac562a231c7cbfc923d2deae9f5d21064eaf1fb924cc824bbca7de0938e3c52"} Oct 30 01:58:16.243925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:16.243537 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:16.285177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:16.285087 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" podStartSLOduration=2.742588533 podStartE2EDuration="4.285072512s" totalImagesPullingTime="1.542483979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:16.281861564 +0000 UTC m=+1724.673669200" watchObservedRunningTime="2026-10-30 01:58:16.285072512 +0000 UTC m=+1724.676880145" Oct 30 01:58:16.697536 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:16.697502 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:58:17.130704 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.130671 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:17.252949 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.252919 6275 generic.go:356] "Generic (PLEG): container finished" podID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerID="e47d392c51964d92dcc1a03d63a59999559af40f52ac9950eb3d121abadb37ad" exitCode=0 Oct 30 01:58:17.253393 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.253009 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" event={"ID":"783ef77e-9b40-4eb0-8062-e35f4446011c","Type":"ContainerDied","Data":"e47d392c51964d92dcc1a03d63a59999559af40f52ac9950eb3d121abadb37ad"} Oct 30 01:58:17.253393 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.253069 6275 scope.go:122] "RemoveContainer" containerID="473665835c2b288b68e1b19f4889564031d9bed0b8b6a8c5e2350288926a046f" Oct 30 01:58:17.280684 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.280662 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:58:17.375512 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375470 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375553 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375583 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375623 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375726 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375671 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375741 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375774 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375797 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:17.375971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375838 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375866 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy\") pod \"783ef77e-9b40-4eb0-8062-e35f4446011c\" (UID: \"783ef77e-9b40-4eb0-8062-e35f4446011c\") " Oct 30 01:58:17.375971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.375926 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:17.376295 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376230 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:17.376345 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376294 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:17.376438 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376403 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:17.376727 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376672 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.376727 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376695 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.376727 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376710 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.376727 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376723 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783ef77e-9b40-4eb0-8062-e35f4446011c-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.377033 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.376736 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.378485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.378451 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:17.378609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.378589 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "kube-api-access-69fkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:17.378652 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.378635 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:17.378755 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.378734 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo" pod "783ef77e-9b40-4eb0-8062-e35f4446011c" (UID: "783ef77e-9b40-4eb0-8062-e35f4446011c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:58:17.478638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.478598 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.478638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.478633 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.478638 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.478642 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69fkp\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-kube-api-access-69fkp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.478882 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:17.478652 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783ef77e-9b40-4eb0-8062-e35f4446011c-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.138699 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:18.138671 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:18.261287 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:18.261259 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" Oct 30 01:58:18.261287 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:18.261278 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4" event={"ID":"783ef77e-9b40-4eb0-8062-e35f4446011c","Type":"ContainerDied","Data":"72cebe001c21d2e8e46eb3e93d650c8f1b2727c09998eebbad5270b537748a2c"} Oct 30 01:58:18.261803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:18.261334 6275 scope.go:122] "RemoveContainer" containerID="e47d392c51964d92dcc1a03d63a59999559af40f52ac9950eb3d121abadb37ad" Oct 30 01:58:18.341205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:18.341167 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4"] Oct 30 01:58:18.345107 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:18.345072 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c93367-openshift-default-7db9fc476c-khlq4"] Oct 30 01:58:19.148015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:19.147976 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g"] Oct 30 01:58:19.742306 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:19.742272 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" path="/var/lib/kubelet/pods/783ef77e-9b40-4eb0-8062-e35f4446011c/volumes" Oct 30 01:58:20.370742 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.370709 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z"] Oct 30 01:58:20.371410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.371386 6275 cpu_manager.go:414] "containerMap: removing container" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" Oct 30 01:58:20.371410 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.371404 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" Oct 30 01:58:20.371612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.371420 6275 cpu_manager.go:414] "containerMap: removing container" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" Oct 30 01:58:20.371612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.371425 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" Oct 30 01:58:20.371612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.371554 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" Oct 30 01:58:20.371612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.371564 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="783ef77e-9b40-4eb0-8062-e35f4446011c" containerName="istio-proxy" Oct 30 01:58:20.376480 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.376460 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.395686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.395659 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7542fa-serving-cert\"" Oct 30 01:58:20.408451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.408421 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z"] Oct 30 01:58:20.445354 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.445321 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.445554 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.445387 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.445634 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.445545 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xkk\" (UniqueName: \"kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.445696 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.445672 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.572155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.557143 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.572155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.557255 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.572155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.557314 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xkk\" (UniqueName: \"kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.572155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.557397 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.572155 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.570486 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.588141 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.587463 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.588141 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.587692 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.642199 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.624827 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xkk\" (UniqueName: \"kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk\") pod \"maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:20.694439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:20.694401 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:21.009411 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:21.009383 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z"] Oct 30 01:58:21.011216 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:21.011187 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebfb1cb_be63_4866_98f7_81d3f488ce91.slice/crio-eae3804dff0e69be6280093a1b20aa006a0b154a9b0d79e7b46a22e759e7174a WatchSource:0}: Error finding container eae3804dff0e69be6280093a1b20aa006a0b154a9b0d79e7b46a22e759e7174a: Status 404 returned error can't find the container with id eae3804dff0e69be6280093a1b20aa006a0b154a9b0d79e7b46a22e759e7174a Oct 30 01:58:21.296975 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:21.296922 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" event={"ID":"3ebfb1cb-be63-4866-98f7-81d3f488ce91","Type":"ContainerStarted","Data":"eae3804dff0e69be6280093a1b20aa006a0b154a9b0d79e7b46a22e759e7174a"} Oct 30 01:58:22.305904 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:22.305845 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" event={"ID":"3ebfb1cb-be63-4866-98f7-81d3f488ce91","Type":"ContainerStarted","Data":"76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912"} Oct 30 01:58:22.306354 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:22.305934 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:22.349049 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:22.348990 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" podStartSLOduration=1.938810734 podStartE2EDuration="2.348976016s" totalImagesPullingTime="410.165282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:22.345395707 +0000 UTC m=+1730.737203352" watchObservedRunningTime="2026-10-30 01:58:22.348976016 +0000 UTC m=+1730.740783667" Oct 30 01:58:24.159527 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.157818 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" podUID="356d78a7-e84d-4e78-b7bf-b14406f2243b" containerName="payload-processing" containerID="cri-o://6e97a0eb9791054b21649a082fee50ad60bc592fada7ea1de2e10d58940cd633" gracePeriod=25 Oct 30 01:58:24.323443 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.323408 6275 generic.go:356] "Generic (PLEG): container finished" podID="356d78a7-e84d-4e78-b7bf-b14406f2243b" containerID="6e97a0eb9791054b21649a082fee50ad60bc592fada7ea1de2e10d58940cd633" exitCode=0 Oct 30 01:58:24.323623 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.323490 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" event={"ID":"356d78a7-e84d-4e78-b7bf-b14406f2243b","Type":"ContainerDied","Data":"6e97a0eb9791054b21649a082fee50ad60bc592fada7ea1de2e10d58940cd633"} Oct 30 01:58:24.472305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.472279 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:58:24.564557 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.564509 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q\" (UniqueName: \"kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q\") pod \"356d78a7-e84d-4e78-b7bf-b14406f2243b\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " Oct 30 01:58:24.564762 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.564715 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume\") pod \"356d78a7-e84d-4e78-b7bf-b14406f2243b\" (UID: \"356d78a7-e84d-4e78-b7bf-b14406f2243b\") " Oct 30 01:58:24.565190 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.565162 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume" pod "356d78a7-e84d-4e78-b7bf-b14406f2243b" (UID: "356d78a7-e84d-4e78-b7bf-b14406f2243b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:24.565690 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.565639 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356d78a7-e84d-4e78-b7bf-b14406f2243b-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:24.567500 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.567464 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q" pod "356d78a7-e84d-4e78-b7bf-b14406f2243b" (UID: "356d78a7-e84d-4e78-b7bf-b14406f2243b"). InnerVolumeSpecName "kube-api-access-qj47q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:24.667469 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:24.667429 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj47q\" (UniqueName: \"kubernetes.io/projected/356d78a7-e84d-4e78-b7bf-b14406f2243b-kube-api-access-qj47q\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.341337 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:25.341299 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" Oct 30 01:58:25.341337 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:25.341320 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g" event={"ID":"356d78a7-e84d-4e78-b7bf-b14406f2243b","Type":"ContainerDied","Data":"b5fd7a0f0ea409a6598e359e47ce465ed19490e35f180112d47e42a1975a9d0e"} Oct 30 01:58:25.341902 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:25.341424 6275 scope.go:122] "RemoveContainer" containerID="6e97a0eb9791054b21649a082fee50ad60bc592fada7ea1de2e10d58940cd633" Oct 30 01:58:25.392324 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:25.392288 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g"] Oct 30 01:58:25.401523 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:25.401495 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2818a6-6fc6bff7c6-vx86g"] Oct 30 01:58:25.812146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:25.803074 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356d78a7-e84d-4e78-b7bf-b14406f2243b" path="/var/lib/kubelet/pods/356d78a7-e84d-4e78-b7bf-b14406f2243b/volumes" Oct 30 01:58:27.255690 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:27.255659 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 01:58:28.318454 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:28.318378 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 01:58:30.810830 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.810792 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk"] Oct 30 01:58:30.811996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.811969 6275 cpu_manager.go:414] "containerMap: removing container" podUID="356d78a7-e84d-4e78-b7bf-b14406f2243b" containerName="payload-processing" Oct 30 01:58:30.812092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.811997 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356d78a7-e84d-4e78-b7bf-b14406f2243b" containerName="payload-processing" Oct 30 01:58:30.812224 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.812205 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="356d78a7-e84d-4e78-b7bf-b14406f2243b" containerName="payload-processing" Oct 30 01:58:30.817932 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.817908 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.833052 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.833023 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk"] Oct 30 01:58:30.840242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.840221 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86dd79-openshift-default-dockercfg-4zq9k\"" Oct 30 01:58:30.897181 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897145 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897257 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9css\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897305 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897322 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897466 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897580 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897564 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897649 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897711 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897689 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:30.897766 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:30.897753 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.010561 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.010519 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.030085 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.017820 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.047989 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.032087 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.048367 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.048335 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9css\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.048556 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.048539 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.049919 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.049894 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.050426 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.050404 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.050724 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.050705 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.050909 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.050891 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.051440 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.051420 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.052148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.011584 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.052286 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.020547 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.052387 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.036648 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.052478 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.052085 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.065335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.059927 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.076850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.076810 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.092495 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.092461 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9css\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.098807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.098565 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.140693 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.140417 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:31.649925 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.649900 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk"] Oct 30 01:58:31.653864 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:31.653829 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775c7e9b_b1d5_4617_9359_6da3cb9e752c.slice/crio-0cfc108880c63ac7a0cc58e39f9dbf87bb33826006fa8536f46e32a3ee1d9b34 WatchSource:0}: Error finding container 0cfc108880c63ac7a0cc58e39f9dbf87bb33826006fa8536f46e32a3ee1d9b34: Status 404 returned error can't find the container with id 0cfc108880c63ac7a0cc58e39f9dbf87bb33826006fa8536f46e32a3ee1d9b34 Oct 30 01:58:31.656644 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.656607 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:31.656743 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.656679 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:31.656743 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:31.656711 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:32.406921 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:32.406882 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" event={"ID":"775c7e9b-b1d5-4617-9359-6da3cb9e752c","Type":"ContainerStarted","Data":"da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88"} Oct 30 01:58:32.407444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:32.406941 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" event={"ID":"775c7e9b-b1d5-4617-9359-6da3cb9e752c","Type":"ContainerStarted","Data":"0cfc108880c63ac7a0cc58e39f9dbf87bb33826006fa8536f46e32a3ee1d9b34"} Oct 30 01:58:32.468408 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:32.468350 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podStartSLOduration=2.468333433 podStartE2EDuration="2.468333433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:32.465561512 +0000 UTC m=+1740.857369145" watchObservedRunningTime="2026-10-30 01:58:32.468333433 +0000 UTC m=+1740.860141143" Oct 30 01:58:33.143203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:33.143167 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:33.148859 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:33.148832 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:33.518698 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:33.518644 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:33.564145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:33.561251 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:58:37.106532 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.106486 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl"] Oct 30 01:58:37.113189 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.113163 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.127958 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.127930 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86dd79\"" Oct 30 01:58:37.128151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.127930 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86dd79-dockercfg-cxn4v\"" Oct 30 01:58:37.132219 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.132192 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl"] Oct 30 01:58:37.173078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.173026 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.173745 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.173582 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br46t\" (UniqueName: \"kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.275219 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.275179 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.278277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.276215 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br46t\" (UniqueName: \"kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.278277 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.276629 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.433195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.352769 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br46t\" (UniqueName: \"kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.433195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.430386 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:37.978293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:37.978267 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl"] Oct 30 01:58:37.980847 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:37.980819 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efc7006_bb88_4576_9938_26c1b6d2e3f9.slice/crio-3e17b5b9c5dbf6d60debfa4021296b035741d4d1abcca9ede49123ac9cdac74f WatchSource:0}: Error finding container 3e17b5b9c5dbf6d60debfa4021296b035741d4d1abcca9ede49123ac9cdac74f: Status 404 returned error can't find the container with id 3e17b5b9c5dbf6d60debfa4021296b035741d4d1abcca9ede49123ac9cdac74f Oct 30 01:58:38.660195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:38.653653 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" event={"ID":"7efc7006-bb88-4576-9938-26c1b6d2e3f9","Type":"ContainerStarted","Data":"275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39"} Oct 30 01:58:38.660195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:38.653718 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" event={"ID":"7efc7006-bb88-4576-9938-26c1b6d2e3f9","Type":"ContainerStarted","Data":"3e17b5b9c5dbf6d60debfa4021296b035741d4d1abcca9ede49123ac9cdac74f"} Oct 30 01:58:38.660195 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:38.654956 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:38.721009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:38.720937 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" podStartSLOduration=1.7209154340000001 podStartE2EDuration="1.720915434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:38.70059858 +0000 UTC m=+1747.092406226" watchObservedRunningTime="2026-10-30 01:58:38.720915434 +0000 UTC m=+1747.112723068" Oct 30 01:58:42.945294 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:42.945251 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc"] Oct 30 01:58:42.978432 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:42.978397 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n"] Oct 30 01:58:45.713218 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:45.713183 6275 generic.go:356] "Generic (PLEG): container finished" podID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerID="be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d" exitCode=0 Oct 30 01:58:45.713631 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:45.713252 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" event={"ID":"6d3afe93-cb6b-473e-85d5-4fae2ed27555","Type":"ContainerDied","Data":"be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d"} Oct 30 01:58:45.713681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:45.713667 6275 scope.go:122] "RemoveContainer" containerID="be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d" Oct 30 01:58:45.715116 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:45.715086 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:45.715215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:45.715168 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:45.715215 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:45.715201 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:46.698937 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:46.698899 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:58:46.723471 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:46.723434 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" event={"ID":"6d3afe93-cb6b-473e-85d5-4fae2ed27555","Type":"ContainerStarted","Data":"39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369"} Oct 30 01:58:47.036283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:47.036249 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:47.158146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:47.154455 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:47.166436 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:47.166397 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:47.166632 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:47.166489 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:47.945699 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:47.945647 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" podUID="9cbb3956-0e5c-40b9-b78f-77f1a307148e" containerName="payload-processing" containerID="cri-o://2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743" gracePeriod=25 Oct 30 01:58:47.981607 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:47.980141 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" podUID="5eb21048-65b3-4aad-8431-e1538619a298" containerName="payload-pre-processing" containerID="cri-o://a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a" gracePeriod=25 Oct 30 01:58:48.162305 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:48.158432 6275 logging.go:55] [core] [Channel #1108 SubChannel #1109]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.114:50052", ServerName: "10.134.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.114:50052: connect: connection refused" Oct 30 01:58:48.162305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.160637 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:48.162305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.160688 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:48.494074 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.494038 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:48.584888 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.584851 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume\") pod \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " Oct 30 01:58:48.585281 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.585257 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66\" (UniqueName: \"kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66\") pod \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\" (UID: \"9cbb3956-0e5c-40b9-b78f-77f1a307148e\") " Oct 30 01:58:48.585385 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.585288 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume" pod "9cbb3956-0e5c-40b9-b78f-77f1a307148e" (UID: "9cbb3956-0e5c-40b9-b78f-77f1a307148e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:48.586296 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.586270 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb3956-0e5c-40b9-b78f-77f1a307148e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.589229 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.589190 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66" pod "9cbb3956-0e5c-40b9-b78f-77f1a307148e" (UID: "9cbb3956-0e5c-40b9-b78f-77f1a307148e"). InnerVolumeSpecName "kube-api-access-vzt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:48.600724 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.600699 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:48.699422 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.692865 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume\") pod \"5eb21048-65b3-4aad-8431-e1538619a298\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " Oct 30 01:58:48.699422 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.693035 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm\" (UniqueName: \"kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm\") pod \"5eb21048-65b3-4aad-8431-e1538619a298\" (UID: \"5eb21048-65b3-4aad-8431-e1538619a298\") " Oct 30 01:58:48.699422 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.693720 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzt66\" (UniqueName: \"kubernetes.io/projected/9cbb3956-0e5c-40b9-b78f-77f1a307148e-kube-api-access-vzt66\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.712370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.702694 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm" pod "5eb21048-65b3-4aad-8431-e1538619a298" (UID: "5eb21048-65b3-4aad-8431-e1538619a298"). InnerVolumeSpecName "kube-api-access-5gtfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:48.712370 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.704785 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume" pod "5eb21048-65b3-4aad-8431-e1538619a298" (UID: "5eb21048-65b3-4aad-8431-e1538619a298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:48.799558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.799512 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb21048-65b3-4aad-8431-e1538619a298-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.799558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.799548 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gtfm\" (UniqueName: \"kubernetes.io/projected/5eb21048-65b3-4aad-8431-e1538619a298-kube-api-access-5gtfm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.810481 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.810445 6275 generic.go:356] "Generic (PLEG): container finished" podID="5eb21048-65b3-4aad-8431-e1538619a298" containerID="a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a" exitCode=0 Oct 30 01:58:48.810646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.810510 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" event={"ID":"5eb21048-65b3-4aad-8431-e1538619a298","Type":"ContainerDied","Data":"a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a"} Oct 30 01:58:48.810646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.810563 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" event={"ID":"5eb21048-65b3-4aad-8431-e1538619a298","Type":"ContainerDied","Data":"b7c8d3dad3d209f9bdc864e393bee0fd213cf0855eb21fdd962e2ff9d6cad5f5"} Oct 30 01:58:48.810646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.810586 6275 scope.go:122] "RemoveContainer" containerID="a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a" Oct 30 01:58:48.811192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.811168 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" Oct 30 01:58:48.813203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.813179 6275 generic.go:356] "Generic (PLEG): container finished" podID="9cbb3956-0e5c-40b9-b78f-77f1a307148e" containerID="2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743" exitCode=0 Oct 30 01:58:48.814059 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.814039 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" Oct 30 01:58:48.815234 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.815202 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" event={"ID":"9cbb3956-0e5c-40b9-b78f-77f1a307148e","Type":"ContainerDied","Data":"2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743"} Oct 30 01:58:48.815340 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.815253 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc" event={"ID":"9cbb3956-0e5c-40b9-b78f-77f1a307148e","Type":"ContainerDied","Data":"e97d33c9c2509972328d012ae51b019c7872f58114aacb0952dc22a4422fac3c"} Oct 30 01:58:48.826032 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.826016 6275 scope.go:122] "RemoveContainer" containerID="a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a" Oct 30 01:58:48.826333 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:48.826312 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a\": container with ID starting with a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a not found: ID does not exist" containerID="a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a" Oct 30 01:58:48.826405 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.826340 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a"} err="failed to get container status \"a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a\": rpc error: code = NotFound desc = could not find container \"a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a\": container with ID starting with a90a40aea088f98874604b82565220f675698cbca5f6d1979e38aba13d8ca97a not found: ID does not exist" Oct 30 01:58:48.826405 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.826357 6275 scope.go:122] "RemoveContainer" containerID="2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743" Oct 30 01:58:48.835363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.835344 6275 scope.go:122] "RemoveContainer" containerID="2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743" Oct 30 01:58:48.835637 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:58:48.835612 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743\": container with ID starting with 2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743 not found: ID does not exist" containerID="2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743" Oct 30 01:58:48.835721 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.835647 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743"} err="failed to get container status \"2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743\": rpc error: code = NotFound desc = could not find container \"2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743\": container with ID starting with 2adf8cab0a3c9c221e9fc02289262822d5ca7db770df6ea56382d8b7b857e743 not found: ID does not exist" Oct 30 01:58:48.885984 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.885953 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n"] Oct 30 01:58:48.893453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.893421 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n"] Oct 30 01:58:48.925464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.925430 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc"] Oct 30 01:58:48.929713 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:48.929689 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-56c9884798-4vnfc"] Oct 30 01:58:49.158379 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.158272 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-556c95c5-dls8n" podUID="5eb21048-65b3-4aad-8431-e1538619a298" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.114:50052\" within 1s: context deadline exceeded" Oct 30 01:58:49.164144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.163268 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:49.164144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.163321 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.214890 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js"] Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.215906 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9cbb3956-0e5c-40b9-b78f-77f1a307148e" containerName="payload-processing" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.215929 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbb3956-0e5c-40b9-b78f-77f1a307148e" containerName="payload-processing" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.215957 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5eb21048-65b3-4aad-8431-e1538619a298" containerName="payload-pre-processing" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.215966 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb21048-65b3-4aad-8431-e1538619a298" containerName="payload-pre-processing" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.216193 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbb3956-0e5c-40b9-b78f-77f1a307148e" containerName="payload-processing" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.216215 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb21048-65b3-4aad-8431-e1538619a298" containerName="payload-pre-processing" Oct 30 01:58:49.222562 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.219639 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.261151 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.237685 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-954773-openshift-default-dockercfg-nszsq\"" Oct 30 01:58:49.278573 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.278531 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js"] Oct 30 01:58:49.306273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306220 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306312 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306369 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306409 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306437 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306496 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306475 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnk5\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306550 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306598 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.306797 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.306692 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.412598 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.412671 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.412758 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.412867 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.412917 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.412980 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.413021 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.413049 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.413093 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnk5\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.413980 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.414301 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.417412 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.414929 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.438149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.436720 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.457158 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.445590 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.479563 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.469621 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.479563 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.471795 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.534143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.533547 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.552174 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.537356 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnk5\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5\") pod \"e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:49.766916 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.766876 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb21048-65b3-4aad-8431-e1538619a298" path="/var/lib/kubelet/pods/5eb21048-65b3-4aad-8431-e1538619a298/volumes" Oct 30 01:58:49.767558 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.767535 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbb3956-0e5c-40b9-b78f-77f1a307148e" path="/var/lib/kubelet/pods/9cbb3956-0e5c-40b9-b78f-77f1a307148e/volumes" Oct 30 01:58:49.842145 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:49.841318 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:50.169762 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.169528 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:50.169762 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.169598 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:50.315703 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.315677 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js"] Oct 30 01:58:50.318024 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:50.317992 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0c2dd5_12e1_4fbd_ab38_06abc45df4b4.slice/crio-8c693b337edef2bebe4dbbe607236ed33b28c687989529217bae97ae0a3848a8 WatchSource:0}: Error finding container 8c693b337edef2bebe4dbbe607236ed33b28c687989529217bae97ae0a3848a8: Status 404 returned error can't find the container with id 8c693b337edef2bebe4dbbe607236ed33b28c687989529217bae97ae0a3848a8 Oct 30 01:58:50.321205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.321094 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:50.321205 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.321187 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:50.321321 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.321221 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:58:50.679522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:50.679439 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:58:51.012848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.012814 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" event={"ID":"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4","Type":"ContainerStarted","Data":"70f492442de2b0d163806f289b88bf292ee9024a4edde51847a67c23bfce2c35"} Oct 30 01:58:51.013017 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.012874 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" event={"ID":"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4","Type":"ContainerStarted","Data":"8c693b337edef2bebe4dbbe607236ed33b28c687989529217bae97ae0a3848a8"} Oct 30 01:58:51.068833 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.068718 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podStartSLOduration=2.068698512 podStartE2EDuration="2.068698512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:51.064518609 +0000 UTC m=+1759.456326242" watchObservedRunningTime="2026-10-30 01:58:51.068698512 +0000 UTC m=+1759.460506147" Oct 30 01:58:51.163663 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.163624 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:51.163878 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.163704 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:51.863578 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.863544 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:51.869697 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:51.869663 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:52.018948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:52.018919 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:52.019754 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:52.019734 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:58:52.151920 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:52.151830 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:58:52.151920 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:52.151902 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 01:58:52.670891 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:52.670843 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b"] Oct 30 01:58:52.671444 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:52.671402 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" podUID="fcb7e2ad-a8c8-4765-b623-c97492224810" containerName="maas-api" containerID="cri-o://c5dbaf7e00527a57140d66ea1bdcfe910741c8b64db0d6cc48275de786aea852" gracePeriod=30 Oct 30 01:58:53.138006 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.136107 6275 generic.go:356] "Generic (PLEG): container finished" podID="fcb7e2ad-a8c8-4765-b623-c97492224810" containerID="c5dbaf7e00527a57140d66ea1bdcfe910741c8b64db0d6cc48275de786aea852" exitCode=0 Oct 30 01:58:53.138006 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.137414 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" event={"ID":"fcb7e2ad-a8c8-4765-b623-c97492224810","Type":"ContainerDied","Data":"c5dbaf7e00527a57140d66ea1bdcfe910741c8b64db0d6cc48275de786aea852"} Oct 30 01:58:53.163744 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.163712 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:53.173079 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.173046 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 01:58:53.279015 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.278983 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:58:53.351192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.350578 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca\") pod \"fcb7e2ad-a8c8-4765-b623-c97492224810\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " Oct 30 01:58:53.351192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.350687 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls\") pod \"fcb7e2ad-a8c8-4765-b623-c97492224810\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " Oct 30 01:58:53.351192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.350761 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q\" (UniqueName: \"kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q\") pod \"fcb7e2ad-a8c8-4765-b623-c97492224810\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " Oct 30 01:58:53.351192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.350804 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs\") pod \"fcb7e2ad-a8c8-4765-b623-c97492224810\" (UID: \"fcb7e2ad-a8c8-4765-b623-c97492224810\") " Oct 30 01:58:53.352515 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.352467 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca" pod "fcb7e2ad-a8c8-4765-b623-c97492224810" (UID: "fcb7e2ad-a8c8-4765-b623-c97492224810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:53.366230 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.365639 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs" pod "fcb7e2ad-a8c8-4765-b623-c97492224810" (UID: "fcb7e2ad-a8c8-4765-b623-c97492224810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:53.376709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.373306 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q" pod "fcb7e2ad-a8c8-4765-b623-c97492224810" (UID: "fcb7e2ad-a8c8-4765-b623-c97492224810"). InnerVolumeSpecName "kube-api-access-f7s6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:53.376709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.373427 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls" pod "fcb7e2ad-a8c8-4765-b623-c97492224810" (UID: "fcb7e2ad-a8c8-4765-b623-c97492224810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:53.465933 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.453502 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7s6q\" (UniqueName: \"kubernetes.io/projected/fcb7e2ad-a8c8-4765-b623-c97492224810-kube-api-access-f7s6q\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.465933 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.453539 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.465933 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.453557 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7e2ad-a8c8-4765-b623-c97492224810-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.465933 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.453571 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7e2ad-a8c8-4765-b623-c97492224810-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.680242 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.680163 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf"] Oct 30 01:58:53.680818 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.680804 6275 cpu_manager.go:414] "containerMap: removing container" podUID="fcb7e2ad-a8c8-4765-b623-c97492224810" containerName="maas-api" Oct 30 01:58:53.680865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.680819 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb7e2ad-a8c8-4765-b623-c97492224810" containerName="maas-api" Oct 30 01:58:53.680931 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.680921 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb7e2ad-a8c8-4765-b623-c97492224810" containerName="maas-api" Oct 30 01:58:53.684706 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.684686 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.702761 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.702732 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-86dd79-serving-cert\"" Oct 30 01:58:53.707229 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.707202 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf"] Oct 30 01:58:53.757642 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.757611 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.757812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.757689 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.757812 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.757794 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwgj\" (UniqueName: \"kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.757946 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.757931 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.859513 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.859474 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.859692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.859543 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.859692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.859613 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwgj\" (UniqueName: \"kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.859692 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.859652 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.860267 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.860241 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.862157 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.862114 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.862235 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.862218 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:53.883300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:53.883265 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwgj\" (UniqueName: \"kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj\") pod \"maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:54.009202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.002483 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:54.236340 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.236264 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" event={"ID":"fcb7e2ad-a8c8-4765-b623-c97492224810","Type":"ContainerDied","Data":"326ebc1f142dd566b08b30b04569191109460bbb4e83441a7167578f78f6bc23"} Oct 30 01:58:54.237611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.236841 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b" Oct 30 01:58:54.237611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.236901 6275 scope.go:122] "RemoveContainer" containerID="c5dbaf7e00527a57140d66ea1bdcfe910741c8b64db0d6cc48275de786aea852" Oct 30 01:58:54.300955 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.300856 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b"] Oct 30 01:58:54.312492 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.301260 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-svn7b"] Oct 30 01:58:54.547635 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:54.547605 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf"] Oct 30 01:58:54.549183 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:58:54.549153 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078b1d72_ec54_4f0a_85e5_973b3b4bc06f.slice/crio-e243ffd5605933028a6fc7ce58b39ba01dab7c737d789b820f482abfae93ef2c WatchSource:0}: Error finding container e243ffd5605933028a6fc7ce58b39ba01dab7c737d789b820f482abfae93ef2c: Status 404 returned error can't find the container with id e243ffd5605933028a6fc7ce58b39ba01dab7c737d789b820f482abfae93ef2c Oct 30 01:58:55.382404 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:55.382372 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" event={"ID":"078b1d72-ec54-4f0a-85e5-973b3b4bc06f","Type":"ContainerStarted","Data":"35bc9142f3e01a1b26cccf94e9fda2563be527cfd0fe8c1d0457fac58a006872"} Oct 30 01:58:55.382865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:55.382843 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" event={"ID":"078b1d72-ec54-4f0a-85e5-973b3b4bc06f","Type":"ContainerStarted","Data":"e243ffd5605933028a6fc7ce58b39ba01dab7c737d789b820f482abfae93ef2c"} Oct 30 01:58:55.384462 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:55.383982 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:58:55.425279 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:55.425098 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" podStartSLOduration=1.9889663720000001 podStartE2EDuration="2.425083065s" totalImagesPullingTime="436.116693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:55.424352586 +0000 UTC m=+1763.816160216" watchObservedRunningTime="2026-10-30 01:58:55.425083065 +0000 UTC m=+1763.816890697" Oct 30 01:58:55.742807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:58:55.742727 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb7e2ad-a8c8-4765-b623-c97492224810" path="/var/lib/kubelet/pods/fcb7e2ad-a8c8-4765-b623-c97492224810/volumes" Oct 30 01:59:00.046225 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.046190 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g"] Oct 30 01:59:00.051179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.051117 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.078720 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.078687 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7542fa-dockercfg-bhh9x\"" Oct 30 01:59:00.081568 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.081540 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7542fa\"" Oct 30 01:59:00.082698 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.082674 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g"] Oct 30 01:59:00.147474 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.147438 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrxf\" (UniqueName: \"kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf\") pod \"payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.147672 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.147576 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.249156 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.249087 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.249380 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.249365 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrxf\" (UniqueName: \"kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf\") pod \"payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.249914 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.249888 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.272305 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.272268 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrxf\" (UniqueName: \"kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf\") pod \"payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.364968 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.364874 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:00.670739 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:00.670713 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g"] Oct 30 01:59:00.676640 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:59:00.676454 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162eafa6_0246_407f_8a94_8a152e65a4a8.slice/crio-4646fb66aae547e64aef2283f806959299d5dd1d2b50bfb16666cd133fb89e21 WatchSource:0}: Error finding container 4646fb66aae547e64aef2283f806959299d5dd1d2b50bfb16666cd133fb89e21: Status 404 returned error can't find the container with id 4646fb66aae547e64aef2283f806959299d5dd1d2b50bfb16666cd133fb89e21 Oct 30 01:59:01.561273 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:01.561235 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" event={"ID":"162eafa6-0246-407f-8a94-8a152e65a4a8","Type":"ContainerStarted","Data":"2c79e41286d319d7095d0963f4527f629767913771b91efe0c7ffeee08209385"} Oct 30 01:59:01.561671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:01.561302 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:01.561671 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:01.561327 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" event={"ID":"162eafa6-0246-407f-8a94-8a152e65a4a8","Type":"ContainerStarted","Data":"4646fb66aae547e64aef2283f806959299d5dd1d2b50bfb16666cd133fb89e21"} Oct 30 01:59:01.601591 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:01.601525 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" podStartSLOduration=1.60150785 podStartE2EDuration="1.60150785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:01.599094608 +0000 UTC m=+1769.990902245" watchObservedRunningTime="2026-10-30 01:59:01.60150785 +0000 UTC m=+1769.993315483" Oct 30 01:59:02.476000 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:02.475968 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 01:59:03.151646 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:03.151608 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2"] Oct 30 01:59:03.152353 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:03.151872 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" containerID="cri-o://225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e" gracePeriod=30 Oct 30 01:59:07.527609 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:07.527570 6275 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:59:07.528091 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:07.527649 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:59:08.519830 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.519800 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:59:08.595686 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.595648 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.595707 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.595806 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596032 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.596172 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596148 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.596382 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596276 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596382 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596306 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596382 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596369 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596469 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596449 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596500 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596469 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.596500 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.596493 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket\") pod \"73873853-6186-40f3-bb4d-36aa727df94a\" (UID: \"73873853-6186-40f3-bb4d-36aa727df94a\") " Oct 30 01:59:08.597203 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.597179 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.597369 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.597356 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.597453 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.597288 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.597743 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.597723 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.598593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.598554 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:08.598593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.598563 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:08.598865 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.598827 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.598945 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.598867 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:08.599085 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.599063 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6" pod "73873853-6186-40f3-bb4d-36aa727df94a" (UID: "73873853-6186-40f3-bb4d-36aa727df94a"). InnerVolumeSpecName "kube-api-access-bjrc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:08.636661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.636618 6275 generic.go:356] "Generic (PLEG): container finished" podID="73873853-6186-40f3-bb4d-36aa727df94a" containerID="225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e" exitCode=0 Oct 30 01:59:08.636852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.636681 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" Oct 30 01:59:08.636852 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.636778 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" event={"ID":"73873853-6186-40f3-bb4d-36aa727df94a","Type":"ContainerDied","Data":"225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e"} Oct 30 01:59:08.636954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.636863 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2" event={"ID":"73873853-6186-40f3-bb4d-36aa727df94a","Type":"ContainerDied","Data":"c66be585900a79e3786b156d08232fd47d88570cf0276f09154051d83cfa1142"} Oct 30 01:59:08.636954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.636895 6275 scope.go:122] "RemoveContainer" containerID="225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e" Oct 30 01:59:08.687348 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.687278 6275 scope.go:122] "RemoveContainer" containerID="6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699202 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873853-6186-40f3-bb4d-36aa727df94a-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699226 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699235 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873853-6186-40f3-bb4d-36aa727df94a-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699242 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699250 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjrc6\" (UniqueName: \"kubernetes.io/projected/73873853-6186-40f3-bb4d-36aa727df94a-kube-api-access-bjrc6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699260 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.700274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.699268 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873853-6186-40f3-bb4d-36aa727df94a-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.703084 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.702774 6275 scope.go:122] "RemoveContainer" containerID="225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e" Oct 30 01:59:08.703223 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:08.703194 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e\": container with ID starting with 225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e not found: ID does not exist" containerID="225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e" Oct 30 01:59:08.703310 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.703233 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e"} err="failed to get container status \"225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e\": rpc error: code = NotFound desc = could not find container \"225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e\": container with ID starting with 225abe10672e89d98efa787c6c81cf1d704c931a8aea88f0498a19ae0d647b5e not found: ID does not exist" Oct 30 01:59:08.703310 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.703260 6275 scope.go:122] "RemoveContainer" containerID="6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0" Oct 30 01:59:08.703611 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:08.703590 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0\": container with ID starting with 6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0 not found: ID does not exist" containerID="6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0" Oct 30 01:59:08.703685 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.703628 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0"} err="failed to get container status \"6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0\": rpc error: code = NotFound desc = could not find container \"6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0\": container with ID starting with 6614925d53784e62b1ccefd5bb037b45c0f536b238341d4a24e21741844760a0 not found: ID does not exist" Oct 30 01:59:08.727581 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.727439 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2"] Oct 30 01:59:08.732561 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:08.732526 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-55ck2"] Oct 30 01:59:09.765168 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:09.765039 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73873853-6186-40f3-bb4d-36aa727df94a" path="/var/lib/kubelet/pods/73873853-6186-40f3-bb4d-36aa727df94a/volumes" Oct 30 01:59:12.572280 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:12.572252 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 01:59:15.514091 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.514049 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r"] Oct 30 01:59:15.516980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.515182 6275 cpu_manager.go:414] "containerMap: removing container" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" Oct 30 01:59:15.516980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.515204 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" Oct 30 01:59:15.516980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.515409 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" Oct 30 01:59:15.516980 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.515430 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" Oct 30 01:59:15.522640 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.522073 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.551251 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.551210 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6ba1d9-openshift-default-dockercfg-67skz\"" Oct 30 01:59:15.556272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.556232 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r"] Oct 30 01:59:15.579272 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579232 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.579715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579346 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.579715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579370 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.579715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579504 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.579715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579557 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.579715 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579648 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf47m\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.580101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579722 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.580101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579801 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.580101 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.579833 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801436 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801530 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801581 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801629 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801709 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801744 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801885 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.801975 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.802033 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf47m\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.803576 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.805197 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.804241 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.808213 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.806113 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.808213 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.806468 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.808213 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.807325 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.893007 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.892905 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.893872 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.893816 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.907996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.904947 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf47m\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:15.907996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:15.907070 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:16.034999 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.024276 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b"] Oct 30 01:59:16.036567 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.036533 6275 cpu_manager.go:414] "containerMap: removing container" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" Oct 30 01:59:16.036760 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.036743 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73873853-6186-40f3-bb4d-36aa727df94a" containerName="istio-proxy" Oct 30 01:59:16.043026 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.043001 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.080176 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.080073 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b"] Oct 30 01:59:16.141759 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.141729 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400298 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400397 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400438 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400475 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400569 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400732 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400760 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400795 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m682d\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.401258 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.400842 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.502763 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.502826 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.502870 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m682d\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.502915 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.503038 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.503099 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.503146 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.503180 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.503265 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.503773 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.504415 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.505661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.505301 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.507078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.507047 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.507432 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.507411 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.513535 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.513443 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.515803 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.515768 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.534329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.534263 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.563394 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.561203 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m682d\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.732236 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.731907 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:16.772743 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:16.772604 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:59:16.779617 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.778780 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r"] Oct 30 01:59:16.795497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.790891 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:16.795497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.790984 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:16.795497 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.791056 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:16.923071 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:16.922343 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" event={"ID":"202f1f31-21e9-4309-be30-7807ac06e376","Type":"ContainerStarted","Data":"f03f25452e076244029525e7722402cebfd8aa5ce5061ba771811f3845b60659"} Oct 30 01:59:17.177936 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.177909 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b"] Oct 30 01:59:17.181373 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:59:17.181341 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d77d938_df42_41f9_9194_8ce487f65b05.slice/crio-c48208114ce3f4c53b7e847efac08f2045ff68748bbff34f4a22c29a2fd2fba7 WatchSource:0}: Error finding container c48208114ce3f4c53b7e847efac08f2045ff68748bbff34f4a22c29a2fd2fba7: Status 404 returned error can't find the container with id c48208114ce3f4c53b7e847efac08f2045ff68748bbff34f4a22c29a2fd2fba7 Oct 30 01:59:17.185009 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.184965 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:17.185142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.185060 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:17.185142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.185112 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:17.968377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.968335 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" event={"ID":"202f1f31-21e9-4309-be30-7807ac06e376","Type":"ContainerStarted","Data":"5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5"} Oct 30 01:59:17.970967 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.970932 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" event={"ID":"3d77d938-df42-41f9-9194-8ce487f65b05","Type":"ContainerStarted","Data":"2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd"} Oct 30 01:59:17.971109 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:17.970991 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" event={"ID":"3d77d938-df42-41f9-9194-8ce487f65b05","Type":"ContainerStarted","Data":"c48208114ce3f4c53b7e847efac08f2045ff68748bbff34f4a22c29a2fd2fba7"} Oct 30 01:59:18.023869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.023802 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podStartSLOduration=3.023786602 podStartE2EDuration="3.023786602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:18.020330146 +0000 UTC m=+1786.412137786" watchObservedRunningTime="2026-10-30 01:59:18.023786602 +0000 UTC m=+1786.415594235" Oct 30 01:59:18.066704 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.066625 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podStartSLOduration=3.06660476 podStartE2EDuration="3.06660476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:18.065601041 +0000 UTC m=+1786.457408676" watchObservedRunningTime="2026-10-30 01:59:18.06660476 +0000 UTC m=+1786.458412396" Oct 30 01:59:18.147930 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.147897 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:18.171616 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.171034 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:18.732690 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.732603 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:18.734218 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.734189 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:59:18.734413 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.734392 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:59:18.888012 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.885489 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": read tcp 10.134.0.2:44994->10.134.0.118:15021: read: connection reset by peer" start-of-body= Oct 30 01:59:18.888012 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.885557 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": read tcp 10.134.0.2:44994->10.134.0.118:15021: read: connection reset by peer" Oct 30 01:59:18.978853 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.978820 6275 generic.go:356] "Generic (PLEG): container finished" podID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerID="da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88" exitCode=0 Oct 30 01:59:18.979346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.978898 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" event={"ID":"775c7e9b-b1d5-4617-9359-6da3cb9e752c","Type":"ContainerDied","Data":"da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88"} Oct 30 01:59:18.979346 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.979194 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:18.979757 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.979734 6275 scope.go:122] "RemoveContainer" containerID="da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88" Oct 30 01:59:18.980192 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.980175 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 01:59:18.981162 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.981087 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:18.981300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.981186 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:18.981300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:18.981238 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:19.732778 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:19.732735 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:59:19.732979 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:19.732797 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:59:19.998681 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:19.998590 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" event={"ID":"775c7e9b-b1d5-4617-9359-6da3cb9e752c","Type":"ContainerStarted","Data":"1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576"} Oct 30 01:59:20.150530 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:20.150494 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:59:20.178895 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:20.175272 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:20.182612 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:20.180815 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:20.732998 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:20.732961 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:59:20.733225 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:20.733010 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:59:20.799844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:20.799808 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl"] Oct 30 01:59:21.161355 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:21.161315 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:21.161924 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:21.161890 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:21.738147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:21.733751 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:59:21.738147 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:21.733826 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:59:22.271167 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:22.207452 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:22.271167 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:22.207511 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:22.733650 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:22.733568 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:59:22.733650 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:22.733620 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:59:23.179142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:23.174240 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:23.179142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:23.174314 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:23.733139 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:23.732784 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:59:23.734261 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:23.734192 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:59:24.150062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:24.145550 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:24.150062 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:24.145613 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:24.793086 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:24.793055 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:24.793795 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:24.793770 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:24.794809 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:24.794787 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 01:59:25.164951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.164861 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:25.164951 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.164919 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:25.371709 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.371675 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm"] Oct 30 01:59:25.376096 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.376077 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.395762 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.395729 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm"] Oct 30 01:59:25.395987 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.395865 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c90f5\"" Oct 30 01:59:25.396087 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.395923 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c90f5-dockercfg-ss67g\"" Oct 30 01:59:25.611150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.608922 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247bc\" (UniqueName: \"kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.611150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.609060 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.729220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.711371 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247bc\" (UniqueName: \"kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.746884 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.746851 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.748293 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.748191 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.812099 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.805277 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" podUID="7efc7006-bb88-4576-9938-26c1b6d2e3f9" containerName="payload-pre-processing" containerID="cri-o://275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39" gracePeriod=25 Oct 30 01:59:25.875525 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.865330 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247bc\" (UniqueName: \"kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:25.991374 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:25.991337 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:26.129800 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.129734 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:59:26.165972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.165556 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:59:26.165972 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.165612 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 01:59:26.280587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.279179 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume\") pod \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " Oct 30 01:59:26.280587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.279267 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t\" (UniqueName: \"kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t\") pod \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\" (UID: \"7efc7006-bb88-4576-9938-26c1b6d2e3f9\") " Oct 30 01:59:26.283069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.283019 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume" pod "7efc7006-bb88-4576-9938-26c1b6d2e3f9" (UID: "7efc7006-bb88-4576-9938-26c1b6d2e3f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:26.305143 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.298371 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t" pod "7efc7006-bb88-4576-9938-26c1b6d2e3f9" (UID: "7efc7006-bb88-4576-9938-26c1b6d2e3f9"). InnerVolumeSpecName "kube-api-access-br46t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:26.381603 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.381517 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7efc7006-bb88-4576-9938-26c1b6d2e3f9-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:26.381603 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.381551 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br46t\" (UniqueName: \"kubernetes.io/projected/7efc7006-bb88-4576-9938-26c1b6d2e3f9-kube-api-access-br46t\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:26.394421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.394380 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm"] Oct 30 01:59:26.399377 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:59:26.399346 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf894e4f_fb60_4854_9b1f_4a1fc840d8af.slice/crio-84fe0294e0b973703a8efbf65958252dd9ef3575d8b735a14d8afcd0ae96b57d WatchSource:0}: Error finding container 84fe0294e0b973703a8efbf65958252dd9ef3575d8b735a14d8afcd0ae96b57d: Status 404 returned error can't find the container with id 84fe0294e0b973703a8efbf65958252dd9ef3575d8b735a14d8afcd0ae96b57d Oct 30 01:59:26.589179 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.586728 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" event={"ID":"af894e4f-fb60-4854-9b1f-4a1fc840d8af","Type":"ContainerStarted","Data":"84fe0294e0b973703a8efbf65958252dd9ef3575d8b735a14d8afcd0ae96b57d"} Oct 30 01:59:26.589864 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.589834 6275 generic.go:356] "Generic (PLEG): container finished" podID="7efc7006-bb88-4576-9938-26c1b6d2e3f9" containerID="275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39" exitCode=0 Oct 30 01:59:26.590329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.590311 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" Oct 30 01:59:26.590591 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.590572 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" event={"ID":"7efc7006-bb88-4576-9938-26c1b6d2e3f9","Type":"ContainerDied","Data":"275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39"} Oct 30 01:59:26.590672 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.590609 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl" event={"ID":"7efc7006-bb88-4576-9938-26c1b6d2e3f9","Type":"ContainerDied","Data":"3e17b5b9c5dbf6d60debfa4021296b035741d4d1abcca9ede49123ac9cdac74f"} Oct 30 01:59:26.590672 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.590629 6275 scope.go:122] "RemoveContainer" containerID="275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39" Oct 30 01:59:26.604657 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.604637 6275 scope.go:122] "RemoveContainer" containerID="275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39" Oct 30 01:59:26.604975 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:26.604954 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39\": container with ID starting with 275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39 not found: ID does not exist" containerID="275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39" Oct 30 01:59:26.605040 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.604988 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39"} err="failed to get container status \"275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39\": rpc error: code = NotFound desc = could not find container \"275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39\": container with ID starting with 275769a82eeff6facd1fbc39ae9f65c1c2b9fc5129fde3653053e37f0e990b39 not found: ID does not exist" Oct 30 01:59:26.638239 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.638150 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl"] Oct 30 01:59:26.642309 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:26.642282 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-57c99bdc9f-jl8hl"] Oct 30 01:59:27.147661 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.147624 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:59:27.148112 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.147921 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:59:27.148755 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.148733 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 01:59:27.600673 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.600636 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" event={"ID":"af894e4f-fb60-4854-9b1f-4a1fc840d8af","Type":"ContainerStarted","Data":"0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d"} Oct 30 01:59:27.600863 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.600718 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:27.640229 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.640165 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" podStartSLOduration=2.6401487230000003 podStartE2EDuration="2.640148723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:27.638198222 +0000 UTC m=+1796.030005855" watchObservedRunningTime="2026-10-30 01:59:27.640148723 +0000 UTC m=+1796.031956353" Oct 30 01:59:27.740986 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:27.740949 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efc7006-bb88-4576-9938-26c1b6d2e3f9" path="/var/lib/kubelet/pods/7efc7006-bb88-4576-9938-26c1b6d2e3f9/volumes" Oct 30 01:59:29.690150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:29.683278 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm"] Oct 30 01:59:29.690150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:29.684578 6275 cpu_manager.go:414] "containerMap: removing container" podUID="7efc7006-bb88-4576-9938-26c1b6d2e3f9" containerName="payload-pre-processing" Oct 30 01:59:29.690150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:29.684600 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efc7006-bb88-4576-9938-26c1b6d2e3f9" containerName="payload-pre-processing" Oct 30 01:59:29.690150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:29.684764 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efc7006-bb88-4576-9938-26c1b6d2e3f9" containerName="payload-pre-processing" Oct 30 01:59:29.857188 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:29.851081 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:29.870746 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:29.870633 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b273b6-serving-cert\"" Oct 30 01:59:30.133952 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.133913 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppxs\" (UniqueName: \"kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.134666 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.134637 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.166200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.166160 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.166744 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.166717 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.198516 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.198482 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm"] Oct 30 01:59:30.268288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.268247 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppxs\" (UniqueName: \"kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.268493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.268331 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.268493 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.268360 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.268620 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.268507 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.269139 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.269098 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.270992 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.270961 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.271109 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.270999 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.299522 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.299493 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppxs\" (UniqueName: \"kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs\") pod \"maas-api-e2e-api-a-b273b6-8474756b48-drnkm\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.491510 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.491430 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:30.815844 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.815811 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm"] Oct 30 01:59:30.816568 ip-10-0-143-27 kubenswrapper[6275]: W1030 01:59:30.816539 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92167404_cffd_4c62_a94e_d0e48795ccad.slice/crio-ec7d419d9e1635f21253d839c9528b25b930a18f4e2814b1298c6c61ac25913e WatchSource:0}: Error finding container ec7d419d9e1635f21253d839c9528b25b930a18f4e2814b1298c6c61ac25913e: Status 404 returned error can't find the container with id ec7d419d9e1635f21253d839c9528b25b930a18f4e2814b1298c6c61ac25913e Oct 30 01:59:30.818160 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:30.818114 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:59:31.759138 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:31.759090 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" event={"ID":"92167404-cffd-4c62-a94e-d0e48795ccad","Type":"ContainerStarted","Data":"35cc0d51c9032838d43a63bb3cbe9a0fc27e6e06575df138e2719bd7061fba86"} Oct 30 01:59:31.759292 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:31.759191 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:31.759292 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:31.759209 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" event={"ID":"92167404-cffd-4c62-a94e-d0e48795ccad","Type":"ContainerStarted","Data":"ec7d419d9e1635f21253d839c9528b25b930a18f4e2814b1298c6c61ac25913e"} Oct 30 01:59:31.809729 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:31.809675 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" podStartSLOduration=2.379347735 podStartE2EDuration="2.809661303s" totalImagesPullingTime="430.313568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:31.804959581 +0000 UTC m=+1800.196767214" watchObservedRunningTime="2026-10-30 01:59:31.809661303 +0000 UTC m=+1800.201468936" Oct 30 01:59:32.075821 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:32.075713 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:59:32.077153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:32.077113 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 01:59:33.582934 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:33.582901 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z"] Oct 30 01:59:33.583492 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:33.583219 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" containerID="cri-o://2be506590686af71806082a6f1e41e36cfd4231a387f86d828954989ab79efbe" gracePeriod=30 Oct 30 01:59:37.824202 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:37.824169 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 01:59:38.611064 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.611036 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 01:59:38.881377 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.881286 6275 generic.go:356] "Generic (PLEG): container finished" podID="314bc821-be39-4f39-9fae-949d1c952dd0" containerID="2be506590686af71806082a6f1e41e36cfd4231a387f86d828954989ab79efbe" exitCode=0 Oct 30 01:59:38.881756 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.881362 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" event={"ID":"314bc821-be39-4f39-9fae-949d1c952dd0","Type":"ContainerDied","Data":"2be506590686af71806082a6f1e41e36cfd4231a387f86d828954989ab79efbe"} Oct 30 01:59:38.924326 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.924296 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:59:38.998449 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998416 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.998653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998464 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.998653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998511 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.998653 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998619 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.998848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998692 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.998848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998727 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:38.998848 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998791 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.999014 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998877 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.999014 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998908 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.999014 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998929 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:38.999220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.998989 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:38.999220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999049 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:38.999220 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999091 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo\") pod \"314bc821-be39-4f39-9fae-949d1c952dd0\" (UID: \"314bc821-be39-4f39-9fae-949d1c952dd0\") " Oct 30 01:59:38.999447 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999370 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:38.999948 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999921 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.000102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999952 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.000102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999984 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.000102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:38.999997 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.000102 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.000010 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314bc821-be39-4f39-9fae-949d1c952dd0-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.001850 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.001756 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:39.001947 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.001906 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "kube-api-access-jp8dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:39.002002 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.001945 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:39.002939 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.002911 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo" pod "314bc821-be39-4f39-9fae-949d1c952dd0" (UID: "314bc821-be39-4f39-9fae-949d1c952dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:39.101347 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.101316 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.101347 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.101343 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8dn\" (UniqueName: \"kubernetes.io/projected/314bc821-be39-4f39-9fae-949d1c952dd0-kube-api-access-jp8dn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.101544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.101355 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314bc821-be39-4f39-9fae-949d1c952dd0-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.101544 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.101374 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314bc821-be39-4f39-9fae-949d1c952dd0-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:39.528415 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.528380 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb"] Oct 30 01:59:39.529051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.529037 6275 cpu_manager.go:414] "containerMap: removing container" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" Oct 30 01:59:39.529092 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.529053 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" Oct 30 01:59:39.529236 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.529225 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" containerName="istio-proxy" Oct 30 01:59:39.533111 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.533091 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.568142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.565486 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86dd79\"" Oct 30 01:59:39.568142 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.565598 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86dd79-dockercfg-qdkch\"" Oct 30 01:59:39.569282 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.569176 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb"] Oct 30 01:59:39.612394 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.612360 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.612700 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.612674 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjdr\" (UniqueName: \"kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.714620 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.714570 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjdr\" (UniqueName: \"kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.714796 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.714691 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.715783 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.715749 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.809222 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.803943 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjdr\" (UniqueName: \"kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr\") pod \"payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:39.843182 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.828039 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs"] Oct 30 01:59:39.880146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.870910 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs"] Oct 30 01:59:39.880146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.871032 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:39.887365 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:39.887085 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:40.012283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.011874 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" event={"ID":"314bc821-be39-4f39-9fae-949d1c952dd0","Type":"ContainerDied","Data":"0c6f55618e235aa42b71c58b4c41b1fee2d4f7ce91d78c736e78ad0abe55e665"} Oct 30 01:59:40.012283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.011945 6275 scope.go:122] "RemoveContainer" containerID="2be506590686af71806082a6f1e41e36cfd4231a387f86d828954989ab79efbe" Oct 30 01:59:40.012283 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.012092 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z" Oct 30 01:59:40.044146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.035003 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24k44\" (UniqueName: \"kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44\") pod \"payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.044146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.035303 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.148144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.146762 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24k44\" (UniqueName: \"kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44\") pod \"payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.148144 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.146876 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.148421 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.148227 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.159313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.155205 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z"] Oct 30 01:59:40.159313 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.158253 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cgp5z"] Oct 30 01:59:40.174971 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.172624 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24k44\" (UniqueName: \"kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44\") pod \"payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.211996 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.211965 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:40.695579 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.692473 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb"] Oct 30 01:59:40.701185 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:40.701157 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs"] Oct 30 01:59:41.021194 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.021156 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" event={"ID":"c000114b-e1cd-4948-9465-a95addf7fa39","Type":"ContainerStarted","Data":"306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a"} Oct 30 01:59:41.021670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.021217 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" event={"ID":"c000114b-e1cd-4948-9465-a95addf7fa39","Type":"ContainerStarted","Data":"d774aa944c03154edcba6b4c0a7c01fe0a8be5dc7375b8420f7322426ac53289"} Oct 30 01:59:41.021670 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.021243 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:41.022893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.022865 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" event={"ID":"0f553469-b20f-4c12-ab61-4f6369c3df4f","Type":"ContainerStarted","Data":"a9f0a992f3455d22556f7d39a10ab54c29871480cfaae7ab3fe3ab6e4077311e"} Oct 30 01:59:41.023032 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.022918 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" event={"ID":"0f553469-b20f-4c12-ab61-4f6369c3df4f","Type":"ContainerStarted","Data":"bc2de80c463331b8ef329ee8d252c2cef7bd16f75d2959b0211e459c7a498836"} Oct 30 01:59:41.023032 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.022946 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:41.071042 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.070979 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" podStartSLOduration=2.070961387 podStartE2EDuration="2.070961387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:41.06256183 +0000 UTC m=+1809.454369465" watchObservedRunningTime="2026-10-30 01:59:41.070961387 +0000 UTC m=+1809.462769021" Oct 30 01:59:41.103547 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.103457 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" podStartSLOduration=2.103444405 podStartE2EDuration="2.103444405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:41.100823196 +0000 UTC m=+1809.492630826" watchObservedRunningTime="2026-10-30 01:59:41.103444405 +0000 UTC m=+1809.495252038" Oct 30 01:59:41.775792 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:41.775753 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314bc821-be39-4f39-9fae-949d1c952dd0" path="/var/lib/kubelet/pods/314bc821-be39-4f39-9fae-949d1c952dd0/volumes" Oct 30 01:59:42.103245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.100883 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g"] Oct 30 01:59:42.103245 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.101150 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" containerID="cri-o://dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c" gracePeriod=30 Oct 30 01:59:42.263149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.247176 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4"] Oct 30 01:59:42.263149 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.247561 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" containerID="cri-o://fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f" gracePeriod=30 Oct 30 01:59:42.436645 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.436263 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr"] Oct 30 01:59:42.470153 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.456339 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" containerID="cri-o://e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25" gracePeriod=30 Oct 30 01:59:42.534732 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.534689 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s"] Oct 30 01:59:42.535018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:42.534969 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" containerID="cri-o://602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf" gracePeriod=30 Oct 30 01:59:46.702178 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:46.701837 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 01:59:47.504329 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.504252 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568032 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568172 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568211 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568257 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568393 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568470 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568577 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568632 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.570349 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.568749 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy\") pod \"f03ff625-40fc-4827-a208-b72a2c83bcfe\" (UID: \"f03ff625-40fc-4827-a208-b72a2c83bcfe\") " Oct 30 01:59:47.572553 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.572145 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.573291 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.573163 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.573695 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.573669 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:47.573806 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.573759 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.574004 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.573972 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.576664 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.576632 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:47.577507 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.577469 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.577587 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.577500 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "kube-api-access-hfkkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:47.578439 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.578416 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token" pod "f03ff625-40fc-4827-a208-b72a2c83bcfe" (UID: "f03ff625-40fc-4827-a208-b72a2c83bcfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:47.672288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672248 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672288 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672289 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03ff625-40fc-4827-a208-b72a2c83bcfe-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672305 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672319 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672335 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672349 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfkkv\" (UniqueName: \"kubernetes.io/projected/f03ff625-40fc-4827-a208-b72a2c83bcfe-kube-api-access-hfkkv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672362 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672376 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.672485 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.672390 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03ff625-40fc-4827-a208-b72a2c83bcfe-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.673348 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.673317 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.806837 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.806913 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.806967 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.807026 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.807074 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.807175 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.807279 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.807326 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.808869 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.807438 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6\") pod \"2143f36d-f231-4f4c-bd77-84fd468cfb15\" (UID: \"2143f36d-f231-4f4c-bd77-84fd468cfb15\") " Oct 30 01:59:47.825927 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.825473 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.826907 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.826647 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:47.827504 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.827314 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.827862 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.827672 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.828022 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.827990 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.841725 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.840905 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:47.948233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.918149 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.948233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.918186 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.948233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.918201 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.948233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.918214 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.948233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.918229 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2143f36d-f231-4f4c-bd77-84fd468cfb15-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.948233 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.918241 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:47.958214 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.958163 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:47.981166 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.981109 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "kube-api-access-dd2t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:47.984300 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:47.984266 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo" pod "2143f36d-f231-4f4c-bd77-84fd468cfb15" (UID: "2143f36d-f231-4f4c-bd77-84fd468cfb15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:48.022611 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.019769 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:59:48.029858 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.029501 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.029858 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.029532 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2t6\" (UniqueName: \"kubernetes.io/projected/2143f36d-f231-4f4c-bd77-84fd468cfb15-kube-api-access-dd2t6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.029858 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.029556 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2143f36d-f231-4f4c-bd77-84fd468cfb15-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.037166 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.037138 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:59:48.079494 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.079405 6275 generic.go:356] "Generic (PLEG): container finished" podID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerID="dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c" exitCode=0 Oct 30 01:59:48.079651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.079490 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" Oct 30 01:59:48.079651 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.079493 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" event={"ID":"f03ff625-40fc-4827-a208-b72a2c83bcfe","Type":"ContainerDied","Data":"dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c"} Oct 30 01:59:48.079786 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.079675 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g" event={"ID":"f03ff625-40fc-4827-a208-b72a2c83bcfe","Type":"ContainerDied","Data":"a946284a6de394996d42643e6fe30f9ac00055fb359f816df627a72629f2480b"} Oct 30 01:59:48.079786 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.079712 6275 scope.go:122] "RemoveContainer" containerID="dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c" Oct 30 01:59:48.082997 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.082968 6275 generic.go:356] "Generic (PLEG): container finished" podID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerID="fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f" exitCode=0 Oct 30 01:59:48.083335 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.083309 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" Oct 30 01:59:48.083464 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.083404 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" event={"ID":"2143f36d-f231-4f4c-bd77-84fd468cfb15","Type":"ContainerDied","Data":"fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f"} Oct 30 01:59:48.083537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.083454 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4" event={"ID":"2143f36d-f231-4f4c-bd77-84fd468cfb15","Type":"ContainerDied","Data":"ef83008a73fbfa803689b5829cc6cbb5fac73cb0feaff80b181e6e13d75cd7a4"} Oct 30 01:59:48.092303 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.092276 6275 generic.go:356] "Generic (PLEG): container finished" podID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerID="e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25" exitCode=0 Oct 30 01:59:48.092463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.092326 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" Oct 30 01:59:48.092463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.092356 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" event={"ID":"19718249-1de6-4aad-a8a7-15c3cb5c41b8","Type":"ContainerDied","Data":"e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25"} Oct 30 01:59:48.092463 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.092396 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr" event={"ID":"19718249-1de6-4aad-a8a7-15c3cb5c41b8","Type":"ContainerDied","Data":"e695df7d8ad597e36b6371222bcaf7032774ce16f6dca22c28be1600c9013845"} Oct 30 01:59:48.094051 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.094025 6275 generic.go:356] "Generic (PLEG): container finished" podID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerID="602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf" exitCode=0 Oct 30 01:59:48.094246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.094070 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" event={"ID":"16b8cee0-776d-4ae7-a646-d3060e964bb1","Type":"ContainerDied","Data":"602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf"} Oct 30 01:59:48.094246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.094106 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" event={"ID":"16b8cee0-776d-4ae7-a646-d3060e964bb1","Type":"ContainerDied","Data":"bc832f37f0d88e5312c4724d26dbe623de1a83a6eb1adac572e8a5258102d6fa"} Oct 30 01:59:48.094246 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.094234 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s" Oct 30 01:59:48.098559 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.098542 6275 scope.go:122] "RemoveContainer" containerID="dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c" Oct 30 01:59:48.098864 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:48.098834 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c\": container with ID starting with dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c not found: ID does not exist" containerID="dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c" Oct 30 01:59:48.098957 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.098876 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c"} err="failed to get container status \"dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c\": rpc error: code = NotFound desc = could not find container \"dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c\": container with ID starting with dba036c693eb52ea1ff99c5940c61b40f4b170abca58937dc9a0ae0044924c8c not found: ID does not exist" Oct 30 01:59:48.098957 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.098900 6275 scope.go:122] "RemoveContainer" containerID="fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f" Oct 30 01:59:48.110396 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.110378 6275 scope.go:122] "RemoveContainer" containerID="fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f" Oct 30 01:59:48.110666 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:48.110647 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f\": container with ID starting with fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f not found: ID does not exist" containerID="fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f" Oct 30 01:59:48.110722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.110674 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f"} err="failed to get container status \"fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f\": rpc error: code = NotFound desc = could not find container \"fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f\": container with ID starting with fd2acfeff14d9563ad50c34ecac1792c397725356e2f771c60a4c2ad3a53ab8f not found: ID does not exist" Oct 30 01:59:48.110722 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.110691 6275 scope.go:122] "RemoveContainer" containerID="e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25" Oct 30 01:59:48.126893 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.126870 6275 scope.go:122] "RemoveContainer" containerID="e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25" Oct 30 01:59:48.127394 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:48.127365 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25\": container with ID starting with e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25 not found: ID does not exist" containerID="e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25" Oct 30 01:59:48.127549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.127402 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25"} err="failed to get container status \"e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25\": rpc error: code = NotFound desc = could not find container \"e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25\": container with ID starting with e397b9656487bb46f7d93566d366329f65aae1134e1c435635ef3f5d2e455d25 not found: ID does not exist" Oct 30 01:59:48.127549 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.127423 6275 scope.go:122] "RemoveContainer" containerID="602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf" Oct 30 01:59:48.130484 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130457 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.130605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130511 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.130605 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130567 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.130714 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130612 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.130714 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130661 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.130714 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130696 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.130845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130719 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.130845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130745 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.130845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130803 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.130845 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130828 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.131035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130857 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.131035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130891 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.131035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.130948 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.131035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131000 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.131035 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131028 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.131451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131049 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.131451 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131105 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token\") pod \"16b8cee0-776d-4ae7-a646-d3060e964bb1\" (UID: \"16b8cee0-776d-4ae7-a646-d3060e964bb1\") " Oct 30 01:59:48.131593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131455 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.131593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131512 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.131593 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131571 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:48.131749 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131724 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.131810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131794 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.131810 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131804 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.132020 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.131997 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.132177 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.132108 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.132262 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.132225 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data\") pod \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\" (UID: \"19718249-1de6-4aad-a8a7-15c3cb5c41b8\") " Oct 30 01:59:48.132483 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.132459 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.133024 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133001 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133029 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133045 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133058 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133072 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133085 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133097 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133111 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133110 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133140 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.133801 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.133778 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:48.135967 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.135946 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:48.136400 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.136371 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "kube-api-access-wdkcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:48.136667 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.136649 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:48.139537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.139429 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:48.139537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.139477 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.139537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.139494 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy" pod "19718249-1de6-4aad-a8a7-15c3cb5c41b8" (UID: "19718249-1de6-4aad-a8a7-15c3cb5c41b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:48.139537 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.139501 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "kube-api-access-n7mqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:48.140764 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.140743 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo" pod "16b8cee0-776d-4ae7-a646-d3060e964bb1" (UID: "16b8cee0-776d-4ae7-a646-d3060e964bb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:48.143078 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.143058 6275 scope.go:122] "RemoveContainer" containerID="602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf" Oct 30 01:59:48.143634 ip-10-0-143-27 kubenswrapper[6275]: E1030 01:59:48.143606 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf\": container with ID starting with 602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf not found: ID does not exist" containerID="602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf" Oct 30 01:59:48.143735 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.143643 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf"} err="failed to get container status \"602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf\": rpc error: code = NotFound desc = could not find container \"602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf\": container with ID starting with 602e2f62388b8e2b44e00997ae22fe9ea6f71eda2278a55bda16a4f2f4b61ebf not found: ID does not exist" Oct 30 01:59:48.151479 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.151456 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g"] Oct 30 01:59:48.161356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.161331 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-ch68g"] Oct 30 01:59:48.223954 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.223921 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4"] Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.259149 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-xtvr4"] Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260196 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mqp\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-kube-api-access-n7mqp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260232 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260248 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260263 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260276 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260289 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b8cee0-776d-4ae7-a646-d3060e964bb1-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260301 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdkcl\" (UniqueName: \"kubernetes.io/projected/19718249-1de6-4aad-a8a7-15c3cb5c41b8-kube-api-access-wdkcl\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260314 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b8cee0-776d-4ae7-a646-d3060e964bb1-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.275362 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.260348 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19718249-1de6-4aad-a8a7-15c3cb5c41b8-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 01:59:48.511077 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.511048 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr"] Oct 30 01:59:48.516691 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.516664 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dc6rr"] Oct 30 01:59:48.548363 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.548333 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s"] Oct 30 01:59:48.553632 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:48.553602 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-phk4s"] Oct 30 01:59:49.104449 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.104418 6275 generic.go:356] "Generic (PLEG): container finished" podID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerID="70f492442de2b0d163806f289b88bf292ee9024a4edde51847a67c23bfce2c35" exitCode=0 Oct 30 01:59:49.104880 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.104492 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" event={"ID":"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4","Type":"ContainerDied","Data":"70f492442de2b0d163806f289b88bf292ee9024a4edde51847a67c23bfce2c35"} Oct 30 01:59:49.104969 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.104953 6275 scope.go:122] "RemoveContainer" containerID="70f492442de2b0d163806f289b88bf292ee9024a4edde51847a67c23bfce2c35" Oct 30 01:59:49.106700 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.106666 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:49.106807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.106740 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:49.106807 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.106769 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:59:49.742433 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.742401 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" path="/var/lib/kubelet/pods/16b8cee0-776d-4ae7-a646-d3060e964bb1/volumes" Oct 30 01:59:49.742891 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.742876 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" path="/var/lib/kubelet/pods/19718249-1de6-4aad-a8a7-15c3cb5c41b8/volumes" Oct 30 01:59:49.743356 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.743341 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" path="/var/lib/kubelet/pods/2143f36d-f231-4f4c-bd77-84fd468cfb15/volumes" Oct 30 01:59:49.744010 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:49.743993 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" path="/var/lib/kubelet/pods/f03ff625-40fc-4827-a208-b72a2c83bcfe/volumes" Oct 30 01:59:50.119420 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:50.119391 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" event={"ID":"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4","Type":"ContainerStarted","Data":"142bcfafbf570ba85490d6be3ea9aa12fbe9a026cf6705a20d97e838d67db0a8"} Oct 30 01:59:50.856150 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:50.850673 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:59:50.899249 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:50.896541 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:59:50.899249 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:50.896622 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:59:51.862148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:51.854511 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:59:51.862148 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:51.854576 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:59:52.019069 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:52.019038 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:59:52.032846 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:52.032818 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 01:59:52.033018 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:52.032886 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 01:59:52.863274 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:52.863233 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.863700 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:52.863313 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:59:53.858146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:53.857495 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:59:53.858146 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:53.857575 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:59:54.860552 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:54.860509 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:59:54.861200 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:54.861164 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:59:55.854576 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:55.854544 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 01:59:55.855445 ip-10-0-143-27 kubenswrapper[6275]: I1030 01:59:55.855421 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 02:00:03.358158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.355233 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt"] Oct 30 02:00:03.426183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426066 6275 cpu_manager.go:414] "containerMap: removing container" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" Oct 30 02:00:03.426183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426097 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" Oct 30 02:00:03.426183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426139 6275 cpu_manager.go:414] "containerMap: removing container" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" Oct 30 02:00:03.426183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426149 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" Oct 30 02:00:03.426183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426172 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" Oct 30 02:00:03.426183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426181 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" Oct 30 02:00:03.426580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426228 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" Oct 30 02:00:03.426580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426236 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" Oct 30 02:00:03.426580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426472 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b8cee0-776d-4ae7-a646-d3060e964bb1" containerName="istio-proxy" Oct 30 02:00:03.426580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426498 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="19718249-1de6-4aad-a8a7-15c3cb5c41b8" containerName="istio-proxy" Oct 30 02:00:03.426580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426516 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03ff625-40fc-4827-a208-b72a2c83bcfe" containerName="istio-proxy" Oct 30 02:00:03.426580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.426542 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2143f36d-f231-4f4c-bd77-84fd468cfb15" containerName="istio-proxy" Oct 30 02:00:03.452584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.452545 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.454873 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.454846 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt"] Oct 30 02:00:03.470902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.469599 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-954773\"" Oct 30 02:00:03.482453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.482423 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-954773-dockercfg-rxjgf\"" Oct 30 02:00:03.534232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.534192 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.535156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.535111 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx\") pod \"payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.576690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.576661 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn"] Oct 30 02:00:03.583085 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.582326 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.610866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.610778 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn"] Oct 30 02:00:03.640190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.639528 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn52\" (UniqueName: \"kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52\") pod \"payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.640190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.639598 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.640190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.639655 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.640190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.639780 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx\") pod \"payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.641383 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.641172 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.662729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.662693 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx\") pod \"payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.752540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.751414 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn52\" (UniqueName: \"kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52\") pod \"payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.752540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.751509 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.752540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.752467 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.774221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.774189 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn52\" (UniqueName: \"kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52\") pod \"payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:03.801022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.800988 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:03.923183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:03.923132 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:04.112982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.112958 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt"] Oct 30 02:00:04.115640 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:04.115607 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0bac3f_02d9_44a0_a8cf_b91d9369e61e.slice/crio-56a39bd9ae4cdbd848aa745b88b032d2cdb1643a2eab8f061b65a3ddc493d8f6 WatchSource:0}: Error finding container 56a39bd9ae4cdbd848aa745b88b032d2cdb1643a2eab8f061b65a3ddc493d8f6: Status 404 returned error can't find the container with id 56a39bd9ae4cdbd848aa745b88b032d2cdb1643a2eab8f061b65a3ddc493d8f6 Oct 30 02:00:04.234896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.234871 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn"] Oct 30 02:00:04.237217 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:04.237181 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe92a87_64a0_4d6f_9ef6_06680630a68c.slice/crio-42ca35e445e09859ad9b4224729bd84f755c9c1c344872b609b0b86ca0e67994 WatchSource:0}: Error finding container 42ca35e445e09859ad9b4224729bd84f755c9c1c344872b609b0b86ca0e67994: Status 404 returned error can't find the container with id 42ca35e445e09859ad9b4224729bd84f755c9c1c344872b609b0b86ca0e67994 Oct 30 02:00:04.356050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.356015 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" event={"ID":"efe92a87-64a0-4d6f-9ef6-06680630a68c","Type":"ContainerStarted","Data":"42ca35e445e09859ad9b4224729bd84f755c9c1c344872b609b0b86ca0e67994"} Oct 30 02:00:04.357958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.357922 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" event={"ID":"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e","Type":"ContainerStarted","Data":"11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746"} Oct 30 02:00:04.358077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.357982 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" event={"ID":"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e","Type":"ContainerStarted","Data":"56a39bd9ae4cdbd848aa745b88b032d2cdb1643a2eab8f061b65a3ddc493d8f6"} Oct 30 02:00:04.358077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.358028 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:04.396440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:04.396390 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" podStartSLOduration=1.39637735 podStartE2EDuration="1.39637735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:04.395684319 +0000 UTC m=+1832.787491952" watchObservedRunningTime="2026-10-30 02:00:04.39637735 +0000 UTC m=+1832.788184982" Oct 30 02:00:05.367472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:05.367432 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" event={"ID":"efe92a87-64a0-4d6f-9ef6-06680630a68c","Type":"ContainerStarted","Data":"7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7"} Oct 30 02:00:05.367696 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:05.367508 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:05.407186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:05.407099 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" podStartSLOduration=2.40708099 podStartE2EDuration="2.40708099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:05.405716229 +0000 UTC m=+1833.797523862" watchObservedRunningTime="2026-10-30 02:00:05.40708099 +0000 UTC m=+1833.798888625" Oct 30 02:00:15.370770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:15.370734 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:00:16.377776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:16.377745 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:00:16.713085 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:00:16.712993 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:00:17.642738 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:17.642700 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm"] Oct 30 02:00:18.981600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:18.981564 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:18.982071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:18.981638 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:19.649274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:19.649241 6275 generic.go:356] "Generic (PLEG): container finished" podID="202f1f31-21e9-4309-be30-7807ac06e376" containerID="5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5" exitCode=0 Oct 30 02:00:19.649469 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:19.649303 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" event={"ID":"202f1f31-21e9-4309-be30-7807ac06e376","Type":"ContainerDied","Data":"5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5"} Oct 30 02:00:19.649709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:19.649696 6275 scope.go:122] "RemoveContainer" containerID="5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5" Oct 30 02:00:19.651307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:19.651272 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:00:19.651417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:19.651360 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:00:19.651482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:19.651412 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:00:20.717632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:20.717586 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" event={"ID":"202f1f31-21e9-4309-be30-7807ac06e376","Type":"ContainerStarted","Data":"0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1"} Oct 30 02:00:21.144146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:21.143491 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 02:00:21.144146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:21.143619 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:21.144146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:21.143665 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:22.144935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:22.144654 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:22.145559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:22.145524 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:22.644247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:22.644193 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" podUID="af894e4f-fb60-4854-9b1f-4a1fc840d8af" containerName="payload-pre-processing" containerID="cri-o://0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d" gracePeriod=25 Oct 30 02:00:23.079004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.078972 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 02:00:23.171776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.171740 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:23.172471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.172418 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:23.173259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.173228 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc\" (UniqueName: \"kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc\") pod \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " Oct 30 02:00:23.173386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.173334 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume\") pod \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\" (UID: \"af894e4f-fb60-4854-9b1f-4a1fc840d8af\") " Oct 30 02:00:23.173722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.173700 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume" pod "af894e4f-fb60-4854-9b1f-4a1fc840d8af" (UID: "af894e4f-fb60-4854-9b1f-4a1fc840d8af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.174105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.174082 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af894e4f-fb60-4854-9b1f-4a1fc840d8af-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.175938 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.175912 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc" pod "af894e4f-fb60-4854-9b1f-4a1fc840d8af" (UID: "af894e4f-fb60-4854-9b1f-4a1fc840d8af"). InnerVolumeSpecName "kube-api-access-247bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.275316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.275220 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247bc\" (UniqueName: \"kubernetes.io/projected/af894e4f-fb60-4854-9b1f-4a1fc840d8af-kube-api-access-247bc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.777150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.775725 6275 generic.go:356] "Generic (PLEG): container finished" podID="af894e4f-fb60-4854-9b1f-4a1fc840d8af" containerID="0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d" exitCode=0 Oct 30 02:00:23.777150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.775808 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" event={"ID":"af894e4f-fb60-4854-9b1f-4a1fc840d8af","Type":"ContainerDied","Data":"0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d"} Oct 30 02:00:23.777150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.775862 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" event={"ID":"af894e4f-fb60-4854-9b1f-4a1fc840d8af","Type":"ContainerDied","Data":"84fe0294e0b973703a8efbf65958252dd9ef3575d8b735a14d8afcd0ae96b57d"} Oct 30 02:00:23.777150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.775891 6275 scope.go:122] "RemoveContainer" containerID="0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d" Oct 30 02:00:23.777150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.776065 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm" Oct 30 02:00:23.799676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.799652 6275 scope.go:122] "RemoveContainer" containerID="0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d" Oct 30 02:00:23.799989 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:00:23.799968 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d\": container with ID starting with 0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d not found: ID does not exist" containerID="0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d" Oct 30 02:00:23.800046 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.799999 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d"} err="failed to get container status \"0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d\": rpc error: code = NotFound desc = could not find container \"0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d\": container with ID starting with 0386bfd11e542616cf093a21f12972a9f835099b24aefdd886e9165e11d6f83d not found: ID does not exist" Oct 30 02:00:23.829167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.829137 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm"] Oct 30 02:00:23.864194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:23.864156 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-7ddb795695-mvdkm"] Oct 30 02:00:24.172223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.166195 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:24.172223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.166255 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:24.175788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.175748 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng"] Oct 30 02:00:24.178852 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.178767 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" podUID="56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" containerName="maas-api" containerID="cri-o://bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564" gracePeriod=30 Oct 30 02:00:24.370389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.370350 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c"] Oct 30 02:00:24.370912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.370855 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" podUID="07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" containerName="payload-processing" containerID="cri-o://1ac562a231c7cbfc923d2deae9f5d21064eaf1fb924cc824bbca7de0938e3c52" gracePeriod=30 Oct 30 02:00:24.614483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.614432 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 02:00:24.719214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.716861 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls\") pod \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " Oct 30 02:00:24.719214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.716956 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs\") pod \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " Oct 30 02:00:24.719214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.716995 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca\") pod \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " Oct 30 02:00:24.719214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.717032 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl\" (UniqueName: \"kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl\") pod \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\" (UID: \"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f\") " Oct 30 02:00:24.741191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.719929 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca" pod "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" (UID: "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.741191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.730548 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs" pod "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" (UID: "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:24.773162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.751570 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls" pod "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" (UID: "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:24.787793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.785700 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl" pod "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" (UID: "56adbf72-6b8c-4e60-bb05-be3cef4f4b3f"). InnerVolumeSpecName "kube-api-access-67bkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:24.813066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.813002 6275 generic.go:356] "Generic (PLEG): container finished" podID="07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" containerID="1ac562a231c7cbfc923d2deae9f5d21064eaf1fb924cc824bbca7de0938e3c52" exitCode=0 Oct 30 02:00:24.813253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.813216 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" event={"ID":"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e","Type":"ContainerDied","Data":"1ac562a231c7cbfc923d2deae9f5d21064eaf1fb924cc824bbca7de0938e3c52"} Oct 30 02:00:24.815182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.815151 6275 generic.go:356] "Generic (PLEG): container finished" podID="56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" containerID="bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564" exitCode=0 Oct 30 02:00:24.815327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.815225 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" event={"ID":"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f","Type":"ContainerDied","Data":"bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564"} Oct 30 02:00:24.815327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.815272 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" event={"ID":"56adbf72-6b8c-4e60-bb05-be3cef4f4b3f","Type":"ContainerDied","Data":"831a3977e045a5e6620ace343098b91065de649262d7935f6d602ced559205a1"} Oct 30 02:00:24.815327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.815300 6275 scope.go:122] "RemoveContainer" containerID="bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564" Oct 30 02:00:24.815486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.815418 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng" Oct 30 02:00:24.824529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.824503 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.824682 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.824667 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.824786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.824773 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.824881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.824868 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bkl\" (UniqueName: \"kubernetes.io/projected/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f-kube-api-access-67bkl\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.831075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.831053 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 02:00:24.873793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.873757 6275 scope.go:122] "RemoveContainer" containerID="bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564" Oct 30 02:00:24.878593 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:00:24.878560 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564\": container with ID starting with bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564 not found: ID does not exist" containerID="bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564" Oct 30 02:00:24.878812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.878786 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564"} err="failed to get container status \"bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564\": rpc error: code = NotFound desc = could not find container \"bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564\": container with ID starting with bc7a11f5cda6e74efd9803484040efc41483eb76abfb9d88997e5cc617717564 not found: ID does not exist" Oct 30 02:00:24.910884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.910850 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng"] Oct 30 02:00:24.917367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.917324 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd098e-944fff597-dmbng"] Oct 30 02:00:24.926271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.926244 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume\") pod \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " Oct 30 02:00:24.926421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.926336 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f\" (UniqueName: \"kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f\") pod \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\" (UID: \"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e\") " Oct 30 02:00:24.927390 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.927365 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume" pod "07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" (UID: "07c7bf49-436b-4bcf-b997-d5b3b49fbe8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.929091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.929068 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f" pod "07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" (UID: "07c7bf49-436b-4bcf-b997-d5b3b49fbe8e"). InnerVolumeSpecName "kube-api-access-vz92f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:24.991771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.991691 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr"] Oct 30 02:00:24.992501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992480 6275 cpu_manager.go:414] "containerMap: removing container" podUID="07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" containerName="payload-processing" Oct 30 02:00:24.992501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992500 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" containerName="payload-processing" Oct 30 02:00:24.992681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992518 6275 cpu_manager.go:414] "containerMap: removing container" podUID="af894e4f-fb60-4854-9b1f-4a1fc840d8af" containerName="payload-pre-processing" Oct 30 02:00:24.992681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992526 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af894e4f-fb60-4854-9b1f-4a1fc840d8af" containerName="payload-pre-processing" Oct 30 02:00:24.992681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992608 6275 cpu_manager.go:414] "containerMap: removing container" podUID="56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" containerName="maas-api" Oct 30 02:00:24.992681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992617 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" containerName="maas-api" Oct 30 02:00:24.992882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992819 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" containerName="maas-api" Oct 30 02:00:24.992882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992835 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" containerName="payload-processing" Oct 30 02:00:24.992882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.992846 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="af894e4f-fb60-4854-9b1f-4a1fc840d8af" containerName="payload-pre-processing" Oct 30 02:00:24.997888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:24.997867 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.013478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.013316 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-145336\"" Oct 30 02:00:25.013642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.013480 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-145336-dockercfg-q44ql\"" Oct 30 02:00:25.018258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.015953 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr"] Oct 30 02:00:25.029047 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.029018 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:25.029294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.029273 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz92f\" (UniqueName: \"kubernetes.io/projected/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e-kube-api-access-vz92f\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:25.162145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.145533 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:25.162145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.145607 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:25.162145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.147153 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5757\" (UniqueName: \"kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757\") pod \"payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.162145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.147219 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.249205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.248691 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5757\" (UniqueName: \"kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757\") pod \"payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.249205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.248756 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.281162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.249795 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.393148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.390273 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5757\" (UniqueName: \"kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757\") pod \"payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.631207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.626831 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:25.799300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.790441 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56adbf72-6b8c-4e60-bb05-be3cef4f4b3f" path="/var/lib/kubelet/pods/56adbf72-6b8c-4e60-bb05-be3cef4f4b3f/volumes" Oct 30 02:00:25.799484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:25.799340 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af894e4f-fb60-4854-9b1f-4a1fc840d8af" path="/var/lib/kubelet/pods/af894e4f-fb60-4854-9b1f-4a1fc840d8af/volumes" Oct 30 02:00:26.116678 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.116622 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" event={"ID":"07c7bf49-436b-4bcf-b997-d5b3b49fbe8e","Type":"ContainerDied","Data":"2e4ec5b917260babcb29756b05c0fc389bc853fe194ccc080331efbfb86d5414"} Oct 30 02:00:26.116862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.116717 6275 scope.go:122] "RemoveContainer" containerID="1ac562a231c7cbfc923d2deae9f5d21064eaf1fb924cc824bbca7de0938e3c52" Oct 30 02:00:26.116862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.116859 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c" Oct 30 02:00:26.148449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.148415 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:00:26.149634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.149592 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 02:00:26.176660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.176619 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c"] Oct 30 02:00:26.184030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.183999 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd098e-5d46cf4f46-fqj9c"] Oct 30 02:00:26.194442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:26.194415 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr"] Oct 30 02:00:26.195748 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:26.195726 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92b4bcf_dc7b_448e_bf39_87daf7614ee4.slice/crio-b1e6a60800b9c30ff20a5af4d5ce9b606007d919656e854912cb142262445ba8 WatchSource:0}: Error finding container b1e6a60800b9c30ff20a5af4d5ce9b606007d919656e854912cb142262445ba8: Status 404 returned error can't find the container with id b1e6a60800b9c30ff20a5af4d5ce9b606007d919656e854912cb142262445ba8 Oct 30 02:00:27.131517 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.131477 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" event={"ID":"d92b4bcf-dc7b-448e-bf39-87daf7614ee4","Type":"ContainerStarted","Data":"d1616d4cc25717a1d18c6931e399f54dee2eceea9c62b9163cd39a9eb0798e0e"} Oct 30 02:00:27.132003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.131557 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:27.132003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.131582 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" event={"ID":"d92b4bcf-dc7b-448e-bf39-87daf7614ee4","Type":"ContainerStarted","Data":"b1e6a60800b9c30ff20a5af4d5ce9b606007d919656e854912cb142262445ba8"} Oct 30 02:00:27.147055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.147028 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 02:00:27.147425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.147399 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 02:00:27.148256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.148235 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 02:00:27.173858 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.173809 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" podStartSLOduration=3.173792598 podStartE2EDuration="3.173792598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:27.171107558 +0000 UTC m=+1855.562915192" watchObservedRunningTime="2026-10-30 02:00:27.173792598 +0000 UTC m=+1855.565600232" Oct 30 02:00:27.740337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:27.740306 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c7bf49-436b-4bcf-b997-d5b3b49fbe8e" path="/var/lib/kubelet/pods/07c7bf49-436b-4bcf-b997-d5b3b49fbe8e/volumes" Oct 30 02:00:28.473227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.473186 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn"] Oct 30 02:00:28.489504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.489464 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.514802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.514761 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn"] Oct 30 02:00:28.524532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.520611 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38bc0d\"" Oct 30 02:00:28.524532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.520702 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38bc0d-dockercfg-frkj5\"" Oct 30 02:00:28.632144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.628822 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6s5\" (UniqueName: \"kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5\") pod \"payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.632144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.628990 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.730637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.730561 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6s5\" (UniqueName: \"kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5\") pod \"payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.730940 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.730913 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.731727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.731702 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.754072 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.754033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6s5\" (UniqueName: \"kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5\") pod \"payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:28.872145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:28.861175 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:29.376958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:29.376909 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn"] Oct 30 02:00:29.380395 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:29.380352 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abcc06a_07aa_430c_81a5_31ce5b4e916f.slice/crio-c6e33e06fee41ab4197825c8d59fa2cd77931c4d32c094fdf7db31dd5bcdf3f9 WatchSource:0}: Error finding container c6e33e06fee41ab4197825c8d59fa2cd77931c4d32c094fdf7db31dd5bcdf3f9: Status 404 returned error can't find the container with id c6e33e06fee41ab4197825c8d59fa2cd77931c4d32c094fdf7db31dd5bcdf3f9 Oct 30 02:00:30.196395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:30.196342 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" event={"ID":"6abcc06a-07aa-430c-81a5-31ce5b4e916f","Type":"ContainerStarted","Data":"2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa"} Oct 30 02:00:30.196958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:30.196446 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" event={"ID":"6abcc06a-07aa-430c-81a5-31ce5b4e916f","Type":"ContainerStarted","Data":"c6e33e06fee41ab4197825c8d59fa2cd77931c4d32c094fdf7db31dd5bcdf3f9"} Oct 30 02:00:30.196958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:30.196753 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:30.251689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:30.251622 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" podStartSLOduration=2.25158394 podStartE2EDuration="2.25158394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:30.239388308 +0000 UTC m=+1858.631195941" watchObservedRunningTime="2026-10-30 02:00:30.25158394 +0000 UTC m=+1858.643391573" Oct 30 02:00:32.455538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.455501 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf"] Oct 30 02:00:32.460146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.460107 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.475440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.475413 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-356082-serving-cert\"" Oct 30 02:00:32.489769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.489721 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf"] Oct 30 02:00:32.496812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.496779 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.497029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.496975 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.497146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.497055 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqnkd\" (UniqueName: \"kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.497146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.497108 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.598595 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.598561 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqnkd\" (UniqueName: \"kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.598748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.598639 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.598748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.598733 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.598929 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.598906 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.599421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.599399 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.601289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.601268 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.601371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.601339 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.620396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.620372 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqnkd\" (UniqueName: \"kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd\") pod \"maas-api-e2e-api-b-356082-597b97fb45-v4btf\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:32.782149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:32.778968 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:33.115203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:33.115168 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf"] Oct 30 02:00:33.117068 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:33.117038 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bda3d0_68bb_4aab_958d_4b9f55dbdb94.slice/crio-5828a4eb3a7120f9a4192b20824c2468890350c4221b7fdc3492bbd3b2029f41 WatchSource:0}: Error finding container 5828a4eb3a7120f9a4192b20824c2468890350c4221b7fdc3492bbd3b2029f41: Status 404 returned error can't find the container with id 5828a4eb3a7120f9a4192b20824c2468890350c4221b7fdc3492bbd3b2029f41 Oct 30 02:00:33.220341 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:33.220305 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" event={"ID":"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94","Type":"ContainerStarted","Data":"5828a4eb3a7120f9a4192b20824c2468890350c4221b7fdc3492bbd3b2029f41"} Oct 30 02:00:34.229135 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:34.229025 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" event={"ID":"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94","Type":"ContainerStarted","Data":"c260cd67eadd5caf8dd565f19ba26e407d59145cfc16078c0963085f9960b6fd"} Oct 30 02:00:34.229549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:34.229180 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:34.276880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:34.276823 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" podStartSLOduration=1.490302266 podStartE2EDuration="2.27680937s" totalImagesPullingTime="786.507104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:34.273475663 +0000 UTC m=+1862.665283311" watchObservedRunningTime="2026-10-30 02:00:34.27680937 +0000 UTC m=+1862.668617031" Oct 30 02:00:38.170389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:38.170349 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:00:39.844146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.829564 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v"] Oct 30 02:00:39.844661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.844281 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:39.855148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.845370 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v"] Oct 30 02:00:39.881145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.861283 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph"] Oct 30 02:00:39.915286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.890265 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6ba1d9\"" Oct 30 02:00:40.000242 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.998470 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.000242 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.998526 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdpd\" (UniqueName: \"kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.000242 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:39.998784 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.029243 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.029197 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph"] Oct 30 02:00:40.036167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.034446 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6ba1d9-dockercfg-dxgnv\"" Oct 30 02:00:40.099873 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.099775 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgqv\" (UniqueName: \"kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv\") pod \"payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.100102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.100074 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.100167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.100142 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdpd\" (UniqueName: \"kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.100243 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.100205 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.100927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.100901 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.121926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.121896 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdpd\" (UniqueName: \"kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.201389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.201341 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgqv\" (UniqueName: \"kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv\") pod \"payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.201658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.201637 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.202498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.202477 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.223546 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.223516 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgqv\" (UniqueName: \"kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv\") pod \"payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.241931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.241910 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:00:40.275935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.275902 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:40.358737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.358479 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:40.590862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.590837 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v"] Oct 30 02:00:40.593187 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:40.593149 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb554eb84_c181_48a2_aae5_7dfaab59ab36.slice/crio-39da7ccfce8c774587fc0839105c822729f52b563ab7dea53d5dd03d557cbc23 WatchSource:0}: Error finding container 39da7ccfce8c774587fc0839105c822729f52b563ab7dea53d5dd03d557cbc23: Status 404 returned error can't find the container with id 39da7ccfce8c774587fc0839105c822729f52b563ab7dea53d5dd03d557cbc23 Oct 30 02:00:40.697043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:40.697007 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph"] Oct 30 02:00:40.701956 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:40.701933 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ab53a6_7d7f_4416_9aa6_bcd7cc32052d.slice/crio-e674a4a6ae8af0391a0bbc4b8287858d2dc4176673591e4112397c2c397ddc1d WatchSource:0}: Error finding container e674a4a6ae8af0391a0bbc4b8287858d2dc4176673591e4112397c2c397ddc1d: Status 404 returned error can't find the container with id e674a4a6ae8af0391a0bbc4b8287858d2dc4176673591e4112397c2c397ddc1d Oct 30 02:00:41.207372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.207344 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:00:41.287397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.287354 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" event={"ID":"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d","Type":"ContainerStarted","Data":"4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b"} Oct 30 02:00:41.287596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.287426 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" event={"ID":"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d","Type":"ContainerStarted","Data":"e674a4a6ae8af0391a0bbc4b8287858d2dc4176673591e4112397c2c397ddc1d"} Oct 30 02:00:41.288558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.288533 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:41.291479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.291450 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" event={"ID":"b554eb84-c181-48a2-aae5-7dfaab59ab36","Type":"ContainerStarted","Data":"7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c"} Oct 30 02:00:41.291659 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.291631 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" event={"ID":"b554eb84-c181-48a2-aae5-7dfaab59ab36","Type":"ContainerStarted","Data":"39da7ccfce8c774587fc0839105c822729f52b563ab7dea53d5dd03d557cbc23"} Oct 30 02:00:41.291778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.291675 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:41.327914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.327850 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" podStartSLOduration=2.327825049 podStartE2EDuration="2.327825049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:41.326178716 +0000 UTC m=+1869.717986351" watchObservedRunningTime="2026-10-30 02:00:41.327825049 +0000 UTC m=+1869.719632683" Oct 30 02:00:41.365547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:41.365288 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" podStartSLOduration=2.365269955 podStartE2EDuration="2.365269955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:41.363221443 +0000 UTC m=+1869.755029112" watchObservedRunningTime="2026-10-30 02:00:41.365269955 +0000 UTC m=+1869.757077588" Oct 30 02:00:42.784452 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.784420 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d"] Oct 30 02:00:42.789107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.789088 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:42.813895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.813866 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c90f5-dockercfg-rwf5v\"" Oct 30 02:00:42.814160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.814140 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c90f5\"" Oct 30 02:00:42.819498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.819467 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d"] Oct 30 02:00:42.828136 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.828092 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvk9\" (UniqueName: \"kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:42.828684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.828639 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:42.930178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.930116 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:42.930358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.930258 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvk9\" (UniqueName: \"kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:42.930879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.930861 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:42.951868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:42.951837 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvk9\" (UniqueName: \"kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9\") pod \"payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:43.107673 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.107587 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:43.258610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.258564 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8"] Oct 30 02:00:43.265435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.265405 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.282658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.282634 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8"] Oct 30 02:00:43.337365 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.337324 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.337559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.337432 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwl8\" (UniqueName: \"kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8\") pod \"payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.427284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.427256 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d"] Oct 30 02:00:43.427806 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:43.427778 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4e6f5e_9877_49a9_8fa9_60e20bba0af7.slice/crio-2ea6bea41d5e59488de4ca937f0b83e66f72b5c44aa69d432b2c425e594bda63 WatchSource:0}: Error finding container 2ea6bea41d5e59488de4ca937f0b83e66f72b5c44aa69d432b2c425e594bda63: Status 404 returned error can't find the container with id 2ea6bea41d5e59488de4ca937f0b83e66f72b5c44aa69d432b2c425e594bda63 Oct 30 02:00:43.439678 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.439653 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.439791 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.439695 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwl8\" (UniqueName: \"kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8\") pod \"payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.440448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.440429 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.461721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.461691 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwl8\" (UniqueName: \"kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8\") pod \"payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.583553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.583510 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:43.983512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:43.983482 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8"] Oct 30 02:00:43.991688 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:43.991499 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode181d463_7e41_4707_949a_663cf181149c.slice/crio-c69e75f5a78d3a6dd074eedc05beb12b72a6c286eb7510e5afe263df8b826b47 WatchSource:0}: Error finding container c69e75f5a78d3a6dd074eedc05beb12b72a6c286eb7510e5afe263df8b826b47: Status 404 returned error can't find the container with id c69e75f5a78d3a6dd074eedc05beb12b72a6c286eb7510e5afe263df8b826b47 Oct 30 02:00:44.336321 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.336278 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" event={"ID":"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7","Type":"ContainerStarted","Data":"622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699"} Oct 30 02:00:44.336491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.336348 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" event={"ID":"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7","Type":"ContainerStarted","Data":"2ea6bea41d5e59488de4ca937f0b83e66f72b5c44aa69d432b2c425e594bda63"} Oct 30 02:00:44.336491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.336473 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:44.338294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.338261 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" event={"ID":"e181d463-7e41-4707-949a-663cf181149c","Type":"ContainerStarted","Data":"a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c"} Oct 30 02:00:44.338458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.338316 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" event={"ID":"e181d463-7e41-4707-949a-663cf181149c","Type":"ContainerStarted","Data":"c69e75f5a78d3a6dd074eedc05beb12b72a6c286eb7510e5afe263df8b826b47"} Oct 30 02:00:44.338458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.338344 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:44.381355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.381290 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" podStartSLOduration=2.381274049 podStartE2EDuration="2.381274049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:44.380701521 +0000 UTC m=+1872.772509156" watchObservedRunningTime="2026-10-30 02:00:44.381274049 +0000 UTC m=+1872.773081684" Oct 30 02:00:44.421859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:44.421791 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" podStartSLOduration=1.4217722959999999 podStartE2EDuration="1.421772296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:44.419583223 +0000 UTC m=+1872.811390856" watchObservedRunningTime="2026-10-30 02:00:44.421772296 +0000 UTC m=+1872.813579929" Oct 30 02:00:45.821101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.821065 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4"] Oct 30 02:00:45.825914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.825891 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.852669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.852638 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4"] Oct 30 02:00:45.917320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917285 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917356 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917453 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917473 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzcd\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917492 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917558 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917726 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917849 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:45.917934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:45.917886 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.019581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019541 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.019581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019584 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.019835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019712 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.019835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019738 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.019835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019813 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019851 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzcd\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019879 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019942 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019949 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.019983 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.020111 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.020136 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.020398 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.020694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.020676 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.022079 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.022050 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.022362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.022344 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.042811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.042786 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.043029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.043004 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzcd\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.139868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.139761 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:46.467265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.467235 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4"] Oct 30 02:00:46.469493 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:00:46.469451 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27dab51_2202_47b7_8081_28b6370de6e0.slice/crio-53073857dbabf3d799b7961e761798652776d8d069cab4ba60e8a7fd29ef136d WatchSource:0}: Error finding container 53073857dbabf3d799b7961e761798652776d8d069cab4ba60e8a7fd29ef136d: Status 404 returned error can't find the container with id 53073857dbabf3d799b7961e761798652776d8d069cab4ba60e8a7fd29ef136d Oct 30 02:00:46.472214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.472176 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:00:46.472320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.472244 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:00:46.472320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:46.472276 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:00:46.699656 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:00:46.699618 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:00:47.362896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:47.362857 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" event={"ID":"a27dab51-2202-47b7-8081-28b6370de6e0","Type":"ContainerStarted","Data":"ff95ece34520d73b3a97f26ba32b14f3a52751feb8c265f1bb66211133aaeda2"} Oct 30 02:00:47.363368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:47.362927 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" event={"ID":"a27dab51-2202-47b7-8081-28b6370de6e0","Type":"ContainerStarted","Data":"53073857dbabf3d799b7961e761798652776d8d069cab4ba60e8a7fd29ef136d"} Oct 30 02:00:47.430532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:47.430481 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podStartSLOduration=2.430466786 podStartE2EDuration="2.430466786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:47.417473293 +0000 UTC m=+1875.809280926" watchObservedRunningTime="2026-10-30 02:00:47.430466786 +0000 UTC m=+1875.822274418" Oct 30 02:00:48.166148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:48.165864 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:48.170310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:48.170277 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:00:48.170471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:48.170331 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 02:00:49.140548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:49.140508 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:00:49.141177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:49.141140 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 02:00:50.164158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:50.160080 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:00:50.164158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:50.160168 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 02:00:51.163087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:51.163039 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:00:51.163418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:51.163374 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 02:00:52.140576 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:52.140536 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:00:52.141937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:52.141901 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 02:00:52.321988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:52.321952 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:00:53.159752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:53.159706 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:00:53.160321 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:53.159776 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 02:00:53.328325 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:53.328295 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:00:54.165476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:54.165444 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:54.166743 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:54.166719 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:54.167448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:54.167431 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:00:55.349367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:55.349335 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:00:55.349816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:55.349410 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:00:59.206834 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.206788 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf"] Oct 30 02:00:59.207313 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.207061 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" podUID="078b1d72-ec54-4f0a-85e5-973b3b4bc06f" containerName="maas-api" containerID="cri-o://35bc9142f3e01a1b26cccf94e9fda2563be527cfd0fe8c1d0457fac58a006872" gracePeriod=30 Oct 30 02:00:59.349690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.349481 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs"] Oct 30 02:00:59.349898 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.349761 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" podUID="0f553469-b20f-4c12-ab61-4f6369c3df4f" containerName="payload-processing" containerID="cri-o://a9f0a992f3455d22556f7d39a10ab54c29871480cfaae7ab3fe3ab6e4077311e" gracePeriod=30 Oct 30 02:00:59.373706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.368185 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb"] Oct 30 02:00:59.373706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.368508 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" podUID="c000114b-e1cd-4948-9465-a95addf7fa39" containerName="payload-pre-processing" containerID="cri-o://306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a" gracePeriod=30 Oct 30 02:00:59.397728 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:00:59.397068 6275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f553469_b20f_4c12_ab61_4f6369c3df4f.slice/crio-conmon-a9f0a992f3455d22556f7d39a10ab54c29871480cfaae7ab3fe3ab6e4077311e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:00:59.497432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.497262 6275 generic.go:356] "Generic (PLEG): container finished" podID="0f553469-b20f-4c12-ab61-4f6369c3df4f" containerID="a9f0a992f3455d22556f7d39a10ab54c29871480cfaae7ab3fe3ab6e4077311e" exitCode=0 Oct 30 02:00:59.497432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.497356 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" event={"ID":"0f553469-b20f-4c12-ab61-4f6369c3df4f","Type":"ContainerDied","Data":"a9f0a992f3455d22556f7d39a10ab54c29871480cfaae7ab3fe3ab6e4077311e"} Oct 30 02:00:59.500575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.500402 6275 generic.go:356] "Generic (PLEG): container finished" podID="078b1d72-ec54-4f0a-85e5-973b3b4bc06f" containerID="35bc9142f3e01a1b26cccf94e9fda2563be527cfd0fe8c1d0457fac58a006872" exitCode=0 Oct 30 02:00:59.500575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.500522 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" event={"ID":"078b1d72-ec54-4f0a-85e5-973b3b4bc06f","Type":"ContainerDied","Data":"35bc9142f3e01a1b26cccf94e9fda2563be527cfd0fe8c1d0457fac58a006872"} Oct 30 02:00:59.563956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.563924 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.835752 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls\") pod \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.835875 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca\") pod \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.835984 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs\") pod \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.836146 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj\" (UniqueName: \"kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj\") pod \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\" (UID: \"078b1d72-ec54-4f0a-85e5-973b3b4bc06f\") " Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.840293 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj" pod "078b1d72-ec54-4f0a-85e5-973b3b4bc06f" (UID: "078b1d72-ec54-4f0a-85e5-973b3b4bc06f"). InnerVolumeSpecName "kube-api-access-swwgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.842724 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls" pod "078b1d72-ec54-4f0a-85e5-973b3b4bc06f" (UID: "078b1d72-ec54-4f0a-85e5-973b3b4bc06f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:59.852246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.845029 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs" pod "078b1d72-ec54-4f0a-85e5-973b3b4bc06f" (UID: "078b1d72-ec54-4f0a-85e5-973b3b4bc06f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:59.853775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.853334 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca" pod "078b1d72-ec54-4f0a-85e5-973b3b4bc06f" (UID: "078b1d72-ec54-4f0a-85e5-973b3b4bc06f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:59.967316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.966834 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwgj\" (UniqueName: \"kubernetes.io/projected/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-kube-api-access-swwgj\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:59.967316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.966872 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:59.967316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.966888 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:59.967316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.966903 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078b1d72-ec54-4f0a-85e5-973b3b4bc06f-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:00:59.977292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:00:59.977260 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 02:01:00.036871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.036837 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 02:01:00.068283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.068229 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44\" (UniqueName: \"kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44\") pod \"0f553469-b20f-4c12-ab61-4f6369c3df4f\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " Oct 30 02:01:00.068484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.068454 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume\") pod \"0f553469-b20f-4c12-ab61-4f6369c3df4f\" (UID: \"0f553469-b20f-4c12-ab61-4f6369c3df4f\") " Oct 30 02:01:00.068612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.068594 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr\" (UniqueName: \"kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr\") pod \"c000114b-e1cd-4948-9465-a95addf7fa39\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " Oct 30 02:01:00.068677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.068666 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume\") pod \"c000114b-e1cd-4948-9465-a95addf7fa39\" (UID: \"c000114b-e1cd-4948-9465-a95addf7fa39\") " Oct 30 02:01:00.070110 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.070079 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume" pod "0f553469-b20f-4c12-ab61-4f6369c3df4f" (UID: "0f553469-b20f-4c12-ab61-4f6369c3df4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:00.070333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.070310 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume" pod "c000114b-e1cd-4948-9465-a95addf7fa39" (UID: "c000114b-e1cd-4948-9465-a95addf7fa39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:00.071818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.071786 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44" pod "0f553469-b20f-4c12-ab61-4f6369c3df4f" (UID: "0f553469-b20f-4c12-ab61-4f6369c3df4f"). InnerVolumeSpecName "kube-api-access-24k44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:00.072023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.072004 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr" pod "c000114b-e1cd-4948-9465-a95addf7fa39" (UID: "c000114b-e1cd-4948-9465-a95addf7fa39"). InnerVolumeSpecName "kube-api-access-xrjdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:00.170448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.170414 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjdr\" (UniqueName: \"kubernetes.io/projected/c000114b-e1cd-4948-9465-a95addf7fa39-kube-api-access-xrjdr\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:00.170448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.170443 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c000114b-e1cd-4948-9465-a95addf7fa39-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:00.170448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.170453 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24k44\" (UniqueName: \"kubernetes.io/projected/0f553469-b20f-4c12-ab61-4f6369c3df4f-kube-api-access-24k44\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:00.170692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.170462 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f553469-b20f-4c12-ab61-4f6369c3df4f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:00.508447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.508364 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" Oct 30 02:01:00.508447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.508370 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs" event={"ID":"0f553469-b20f-4c12-ab61-4f6369c3df4f","Type":"ContainerDied","Data":"bc2de80c463331b8ef329ee8d252c2cef7bd16f75d2959b0211e459c7a498836"} Oct 30 02:01:00.509044 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.508507 6275 scope.go:122] "RemoveContainer" containerID="a9f0a992f3455d22556f7d39a10ab54c29871480cfaae7ab3fe3ab6e4077311e" Oct 30 02:01:00.510027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.510010 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" Oct 30 02:01:00.510225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.510050 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf" event={"ID":"078b1d72-ec54-4f0a-85e5-973b3b4bc06f","Type":"ContainerDied","Data":"e243ffd5605933028a6fc7ce58b39ba01dab7c737d789b820f482abfae93ef2c"} Oct 30 02:01:00.511773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.511736 6275 generic.go:356] "Generic (PLEG): container finished" podID="c000114b-e1cd-4948-9465-a95addf7fa39" containerID="306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a" exitCode=0 Oct 30 02:01:00.511911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.511769 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" event={"ID":"c000114b-e1cd-4948-9465-a95addf7fa39","Type":"ContainerDied","Data":"306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a"} Oct 30 02:01:00.511911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.511786 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" Oct 30 02:01:00.511911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.511821 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb" event={"ID":"c000114b-e1cd-4948-9465-a95addf7fa39","Type":"ContainerDied","Data":"d774aa944c03154edcba6b4c0a7c01fe0a8be5dc7375b8420f7322426ac53289"} Oct 30 02:01:00.520727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.520577 6275 scope.go:122] "RemoveContainer" containerID="35bc9142f3e01a1b26cccf94e9fda2563be527cfd0fe8c1d0457fac58a006872" Oct 30 02:01:00.531990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.531970 6275 scope.go:122] "RemoveContainer" containerID="306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a" Oct 30 02:01:00.542048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.542027 6275 scope.go:122] "RemoveContainer" containerID="306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a" Oct 30 02:01:00.542361 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:00.542340 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a\": container with ID starting with 306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a not found: ID does not exist" containerID="306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a" Oct 30 02:01:00.542437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.542376 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a"} err="failed to get container status \"306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a\": rpc error: code = NotFound desc = could not find container \"306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a\": container with ID starting with 306b221241a2423aaa77adc8f4ad10e0f9f94f0daa6c7840dcd1413209df3c5a not found: ID does not exist" Oct 30 02:01:00.582869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.582841 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb"] Oct 30 02:01:00.598797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.595557 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dd79-b646745d7-cqsmb"] Oct 30 02:01:00.624003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.623971 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs"] Oct 30 02:01:00.629029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.629002 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-975c67ddd-9fkvs"] Oct 30 02:01:00.658995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.658967 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf"] Oct 30 02:01:00.663376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:00.663354 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86dd79-76df79cff8-z96tf"] Oct 30 02:01:01.814186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:01.810270 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078b1d72-ec54-4f0a-85e5-973b3b4bc06f" path="/var/lib/kubelet/pods/078b1d72-ec54-4f0a-85e5-973b3b4bc06f/volumes" Oct 30 02:01:01.814186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:01.810984 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f553469-b20f-4c12-ab61-4f6369c3df4f" path="/var/lib/kubelet/pods/0f553469-b20f-4c12-ab61-4f6369c3df4f/volumes" Oct 30 02:01:01.814186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:01.811583 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c000114b-e1cd-4948-9465-a95addf7fa39" path="/var/lib/kubelet/pods/c000114b-e1cd-4948-9465-a95addf7fa39/volumes" Oct 30 02:01:10.058036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:10.057998 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk"] Oct 30 02:01:10.058615 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:10.058298 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" containerID="cri-o://1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576" gracePeriod=30 Oct 30 02:01:10.063223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:10.063197 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b"] Oct 30 02:01:10.063690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:10.063633 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" containerID="cri-o://2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd" gracePeriod=30 Oct 30 02:01:12.150844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:12.150631 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:01:12.150844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:12.150705 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:01:15.423606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.423582 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 02:01:15.426290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.426263 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 02:01:15.528473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528438 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.528651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528493 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.528651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528544 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.528651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528576 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.528651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528616 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.528651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528650 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.528945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528678 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.528945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528722 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.528945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528797 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.528945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528807 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.528945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528895 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.528945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.528920 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.529386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529361 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:15.529500 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529482 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.529578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529488 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.529578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529542 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.529682 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529626 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.529733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529683 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.529787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529763 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.529853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529800 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.529853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529842 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo\") pod \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\" (UID: \"775c7e9b-b1d5-4617-9359-6da3cb9e752c\") " Oct 30 02:01:15.529953 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.529933 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs\") pod \"3d77d938-df42-41f9-9194-8ce487f65b05\" (UID: \"3d77d938-df42-41f9-9194-8ce487f65b05\") " Oct 30 02:01:15.530299 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.530238 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:15.530503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.530450 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.530648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.530623 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.530849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.530826 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.531115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531090 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531137 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531152 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77d938-df42-41f9-9194-8ce487f65b05-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531166 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531180 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531193 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531205 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531217 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.531606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.531421 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.536701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.536643 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.538175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.537271 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "kube-api-access-m682d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.539516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.539479 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:15.541378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.541355 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "kube-api-access-g9css". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.541497 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.541385 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.541559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.541503 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.541616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.541571 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo" pod "3d77d938-df42-41f9-9194-8ce487f65b05" (UID: "3d77d938-df42-41f9-9194-8ce487f65b05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:15.542907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.542865 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.543205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.543151 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token" pod "775c7e9b-b1d5-4617-9359-6da3cb9e752c" (UID: "775c7e9b-b1d5-4617-9359-6da3cb9e752c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.633043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633011 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m682d\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-kube-api-access-m682d\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633041 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633051 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633060 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633069 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77d938-df42-41f9-9194-8ce487f65b05-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633093 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633102 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77d938-df42-41f9-9194-8ce487f65b05-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633110 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9css\" (UniqueName: \"kubernetes.io/projected/775c7e9b-b1d5-4617-9359-6da3cb9e752c-kube-api-access-g9css\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633133 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77d938-df42-41f9-9194-8ce487f65b05-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.633319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.633146 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c7e9b-b1d5-4617-9359-6da3cb9e752c-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.663503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.663468 6275 generic.go:356] "Generic (PLEG): container finished" podID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerID="1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576" exitCode=0 Oct 30 02:01:15.663695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.663530 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" Oct 30 02:01:15.663695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.663595 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" event={"ID":"775c7e9b-b1d5-4617-9359-6da3cb9e752c","Type":"ContainerDied","Data":"1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576"} Oct 30 02:01:15.663695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.663644 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk" event={"ID":"775c7e9b-b1d5-4617-9359-6da3cb9e752c","Type":"ContainerDied","Data":"0cfc108880c63ac7a0cc58e39f9dbf87bb33826006fa8536f46e32a3ee1d9b34"} Oct 30 02:01:15.663695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.663672 6275 scope.go:122] "RemoveContainer" containerID="1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576" Oct 30 02:01:15.665187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.665160 6275 generic.go:356] "Generic (PLEG): container finished" podID="3d77d938-df42-41f9-9194-8ce487f65b05" containerID="2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd" exitCode=0 Oct 30 02:01:15.665187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.665180 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" Oct 30 02:01:15.665348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.665213 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" event={"ID":"3d77d938-df42-41f9-9194-8ce487f65b05","Type":"ContainerDied","Data":"2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd"} Oct 30 02:01:15.665348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.665259 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b" event={"ID":"3d77d938-df42-41f9-9194-8ce487f65b05","Type":"ContainerDied","Data":"c48208114ce3f4c53b7e847efac08f2045ff68748bbff34f4a22c29a2fd2fba7"} Oct 30 02:01:15.675965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.675949 6275 scope.go:122] "RemoveContainer" containerID="da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88" Oct 30 02:01:15.686236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.686218 6275 scope.go:122] "RemoveContainer" containerID="1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576" Oct 30 02:01:15.686530 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:15.686506 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576\": container with ID starting with 1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576 not found: ID does not exist" containerID="1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576" Oct 30 02:01:15.686632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.686535 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576"} err="failed to get container status \"1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576\": rpc error: code = NotFound desc = could not find container \"1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576\": container with ID starting with 1aed0e9f1e72006980eeaf139a4407be9083ee337713da8e972a283664d72576 not found: ID does not exist" Oct 30 02:01:15.686632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.686550 6275 scope.go:122] "RemoveContainer" containerID="da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88" Oct 30 02:01:15.686789 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:15.686772 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88\": container with ID starting with da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88 not found: ID does not exist" containerID="da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88" Oct 30 02:01:15.686845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.686795 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88"} err="failed to get container status \"da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88\": rpc error: code = NotFound desc = could not find container \"da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88\": container with ID starting with da29aae91666eb6870817e22f513df6bed8ca7264db86f1a091692780ead3b88 not found: ID does not exist" Oct 30 02:01:15.686845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.686808 6275 scope.go:122] "RemoveContainer" containerID="2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd" Oct 30 02:01:15.696499 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.696476 6275 scope.go:122] "RemoveContainer" containerID="2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd" Oct 30 02:01:15.696788 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:15.696769 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd\": container with ID starting with 2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd not found: ID does not exist" containerID="2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd" Oct 30 02:01:15.696840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.696794 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd"} err="failed to get container status \"2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd\": rpc error: code = NotFound desc = could not find container \"2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd\": container with ID starting with 2ea4217f7d4e05e7714926898097ccdfa7b9a42d9dabd4529606e89ba150a3dd not found: ID does not exist" Oct 30 02:01:15.726143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.726109 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b"] Oct 30 02:01:15.797934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.797903 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-h9t2b"] Oct 30 02:01:15.822176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.818257 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk"] Oct 30 02:01:15.826292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:15.826245 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-2k7nk"] Oct 30 02:01:16.697769 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:16.697735 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:01:17.740023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.739989 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" path="/var/lib/kubelet/pods/3d77d938-df42-41f9-9194-8ce487f65b05/volumes" Oct 30 02:01:17.740561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.740545 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" path="/var/lib/kubelet/pods/775c7e9b-b1d5-4617-9359-6da3cb9e752c/volumes" Oct 30 02:01:17.816212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816172 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b"] Oct 30 02:01:17.816880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816856 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c000114b-e1cd-4948-9465-a95addf7fa39" containerName="payload-pre-processing" Oct 30 02:01:17.816880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816879 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c000114b-e1cd-4948-9465-a95addf7fa39" containerName="payload-pre-processing" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816912 6275 cpu_manager.go:414] "containerMap: removing container" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816922 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816959 6275 cpu_manager.go:414] "containerMap: removing container" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816967 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816987 6275 cpu_manager.go:414] "containerMap: removing container" podUID="078b1d72-ec54-4f0a-85e5-973b3b4bc06f" containerName="maas-api" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.816992 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078b1d72-ec54-4f0a-85e5-973b3b4bc06f" containerName="maas-api" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817008 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0f553469-b20f-4c12-ab61-4f6369c3df4f" containerName="payload-processing" Oct 30 02:01:17.817073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817013 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f553469-b20f-4c12-ab61-4f6369c3df4f" containerName="payload-processing" Oct 30 02:01:17.817351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817180 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" Oct 30 02:01:17.817351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817192 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d77d938-df42-41f9-9194-8ce487f65b05" containerName="istio-proxy" Oct 30 02:01:17.817351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817202 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="078b1d72-ec54-4f0a-85e5-973b3b4bc06f" containerName="maas-api" Oct 30 02:01:17.817351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817211 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f553469-b20f-4c12-ab61-4f6369c3df4f" containerName="payload-processing" Oct 30 02:01:17.817351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817221 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c000114b-e1cd-4948-9465-a95addf7fa39" containerName="payload-pre-processing" Oct 30 02:01:17.817351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.817229 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" Oct 30 02:01:17.822539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.822521 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:17.836335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.836317 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-95daa2\"" Oct 30 02:01:17.836561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.836547 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-95daa2-dockercfg-wlv9k\"" Oct 30 02:01:17.850899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.850877 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b"] Oct 30 02:01:17.865061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.865034 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7bv\" (UniqueName: \"kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv\") pod \"payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:17.865231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.865105 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:17.959146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.958723 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9"] Oct 30 02:01:17.963144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.960438 6275 cpu_manager.go:414] "containerMap: removing container" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" Oct 30 02:01:17.963144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.960473 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c7e9b-b1d5-4617-9359-6da3cb9e752c" containerName="istio-proxy" Oct 30 02:01:17.988146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.978542 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7bv\" (UniqueName: \"kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv\") pod \"payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:17.988146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.978673 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:17.988146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.981007 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:17.988146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:17.981620 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.044366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.042839 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7bv\" (UniqueName: \"kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv\") pod \"payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:18.089144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.087221 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.089144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.087377 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlwh\" (UniqueName: \"kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh\") pod \"payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.149814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.144977 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:18.197721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.193719 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9"] Oct 30 02:01:18.197721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.194709 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.197721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.194838 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlwh\" (UniqueName: \"kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh\") pod \"payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.200729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.198960 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.228533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.228472 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlwh\" (UniqueName: \"kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh\") pod \"payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.352534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.352450 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:18.526559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.526535 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b"] Oct 30 02:01:18.529575 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:18.529548 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6580c58_9de6_48e8_95fb_cba6698b0cf9.slice/crio-3904941f5f7268c56da59dc74258763dfbd73dbb6dd8aee2bfbbc0ae06cce34d WatchSource:0}: Error finding container 3904941f5f7268c56da59dc74258763dfbd73dbb6dd8aee2bfbbc0ae06cce34d: Status 404 returned error can't find the container with id 3904941f5f7268c56da59dc74258763dfbd73dbb6dd8aee2bfbbc0ae06cce34d Oct 30 02:01:18.680491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.678833 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9"] Oct 30 02:01:18.681352 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:18.681321 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf584bb13_4fdb_4327_814e_5a963a88abcc.slice/crio-76addfb29a9e0c2bf385b05293aaccce49166b491c651af4b878011f67be348a WatchSource:0}: Error finding container 76addfb29a9e0c2bf385b05293aaccce49166b491c651af4b878011f67be348a: Status 404 returned error can't find the container with id 76addfb29a9e0c2bf385b05293aaccce49166b491c651af4b878011f67be348a Oct 30 02:01:18.693448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.693414 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" event={"ID":"e6580c58-9de6-48e8-95fb-cba6698b0cf9","Type":"ContainerStarted","Data":"79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8"} Oct 30 02:01:18.693636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.693609 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:18.693763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.693643 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" event={"ID":"e6580c58-9de6-48e8-95fb-cba6698b0cf9","Type":"ContainerStarted","Data":"3904941f5f7268c56da59dc74258763dfbd73dbb6dd8aee2bfbbc0ae06cce34d"} Oct 30 02:01:18.694939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.694915 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" event={"ID":"f584bb13-4fdb-4327-814e-5a963a88abcc","Type":"ContainerStarted","Data":"76addfb29a9e0c2bf385b05293aaccce49166b491c651af4b878011f67be348a"} Oct 30 02:01:18.747726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:18.747663 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" podStartSLOduration=1.74764377 podStartE2EDuration="1.74764377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:18.740483452 +0000 UTC m=+1907.132291084" watchObservedRunningTime="2026-10-30 02:01:18.74764377 +0000 UTC m=+1907.139451404" Oct 30 02:01:19.747406 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:19.747361 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:19.747800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:19.747698 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" event={"ID":"f584bb13-4fdb-4327-814e-5a963a88abcc","Type":"ContainerStarted","Data":"b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b"} Oct 30 02:01:19.818585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:19.818531 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" podStartSLOduration=2.818516157 podStartE2EDuration="2.818516157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:19.815099261 +0000 UTC m=+1908.206906894" watchObservedRunningTime="2026-10-30 02:01:19.818516157 +0000 UTC m=+1908.210323784" Oct 30 02:01:21.115785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.115747 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm"] Oct 30 02:01:21.116402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.116027 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" podUID="92167404-cffd-4c62-a94e-d0e48795ccad" containerName="maas-api" containerID="cri-o://35cc0d51c9032838d43a63bb3cbe9a0fc27e6e06575df138e2719bd7061fba86" gracePeriod=30 Oct 30 02:01:21.806503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.806356 6275 generic.go:356] "Generic (PLEG): container finished" podID="92167404-cffd-4c62-a94e-d0e48795ccad" containerID="35cc0d51c9032838d43a63bb3cbe9a0fc27e6e06575df138e2719bd7061fba86" exitCode=0 Oct 30 02:01:21.831639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.831600 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" event={"ID":"92167404-cffd-4c62-a94e-d0e48795ccad","Type":"ContainerDied","Data":"35cc0d51c9032838d43a63bb3cbe9a0fc27e6e06575df138e2719bd7061fba86"} Oct 30 02:01:21.832018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.831995 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" event={"ID":"92167404-cffd-4c62-a94e-d0e48795ccad","Type":"ContainerDied","Data":"ec7d419d9e1635f21253d839c9528b25b930a18f4e2814b1298c6c61ac25913e"} Oct 30 02:01:21.832244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.832226 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7d419d9e1635f21253d839c9528b25b930a18f4e2814b1298c6c61ac25913e" Oct 30 02:01:21.844143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.840327 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 02:01:21.900846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.900805 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca\") pod \"92167404-cffd-4c62-a94e-d0e48795ccad\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " Oct 30 02:01:21.901008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.900920 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs\" (UniqueName: \"kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs\") pod \"92167404-cffd-4c62-a94e-d0e48795ccad\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " Oct 30 02:01:21.901054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.901024 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls\") pod \"92167404-cffd-4c62-a94e-d0e48795ccad\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " Oct 30 02:01:21.901090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.901061 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs\") pod \"92167404-cffd-4c62-a94e-d0e48795ccad\" (UID: \"92167404-cffd-4c62-a94e-d0e48795ccad\") " Oct 30 02:01:21.903565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.903537 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca" pod "92167404-cffd-4c62-a94e-d0e48795ccad" (UID: "92167404-cffd-4c62-a94e-d0e48795ccad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:21.905450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.905422 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs" pod "92167404-cffd-4c62-a94e-d0e48795ccad" (UID: "92167404-cffd-4c62-a94e-d0e48795ccad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:21.906536 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.906510 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs" pod "92167404-cffd-4c62-a94e-d0e48795ccad" (UID: "92167404-cffd-4c62-a94e-d0e48795ccad"). InnerVolumeSpecName "kube-api-access-9ppxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:21.907758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:21.907730 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls" pod "92167404-cffd-4c62-a94e-d0e48795ccad" (UID: "92167404-cffd-4c62-a94e-d0e48795ccad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:22.003637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.003603 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:22.003637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.003633 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92167404-cffd-4c62-a94e-d0e48795ccad-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:22.003637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.003643 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92167404-cffd-4c62-a94e-d0e48795ccad-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:22.003894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.003652 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppxs\" (UniqueName: \"kubernetes.io/projected/92167404-cffd-4c62-a94e-d0e48795ccad-kube-api-access-9ppxs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:22.814462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.814423 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm" Oct 30 02:01:22.862702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.862672 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm"] Oct 30 02:01:22.872869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:22.872843 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b273b6-8474756b48-drnkm"] Oct 30 02:01:23.783145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:23.781814 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92167404-cffd-4c62-a94e-d0e48795ccad" path="/var/lib/kubelet/pods/92167404-cffd-4c62-a94e-d0e48795ccad/volumes" Oct 30 02:01:24.619644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:24.619600 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr"] Oct 30 02:01:27.269336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:27.269295 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn"] Oct 30 02:01:29.620503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.620439 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" podUID="d92b4bcf-dc7b-448e-bf39-87daf7614ee4" containerName="payload-pre-processing" containerID="cri-o://d1616d4cc25717a1d18c6931e399f54dee2eceea9c62b9163cd39a9eb0798e0e" gracePeriod=25 Oct 30 02:01:29.748798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.748765 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:01:29.893817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.893784 6275 generic.go:356] "Generic (PLEG): container finished" podID="d92b4bcf-dc7b-448e-bf39-87daf7614ee4" containerID="d1616d4cc25717a1d18c6931e399f54dee2eceea9c62b9163cd39a9eb0798e0e" exitCode=0 Oct 30 02:01:29.893982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.893836 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" event={"ID":"d92b4bcf-dc7b-448e-bf39-87daf7614ee4","Type":"ContainerDied","Data":"d1616d4cc25717a1d18c6931e399f54dee2eceea9c62b9163cd39a9eb0798e0e"} Oct 30 02:01:29.893982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.893880 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" event={"ID":"d92b4bcf-dc7b-448e-bf39-87daf7614ee4","Type":"ContainerDied","Data":"b1e6a60800b9c30ff20a5af4d5ce9b606007d919656e854912cb142262445ba8"} Oct 30 02:01:29.893982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.893895 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e6a60800b9c30ff20a5af4d5ce9b606007d919656e854912cb142262445ba8" Oct 30 02:01:29.907670 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.907644 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:01:29.987293 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.987263 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757\" (UniqueName: \"kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757\") pod \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " Oct 30 02:01:29.987464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.987313 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume\") pod \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\" (UID: \"d92b4bcf-dc7b-448e-bf39-87daf7614ee4\") " Oct 30 02:01:29.987785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.987760 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume" pod "d92b4bcf-dc7b-448e-bf39-87daf7614ee4" (UID: "d92b4bcf-dc7b-448e-bf39-87daf7614ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:29.988064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.988044 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:29.989443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:29.989413 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757" pod "d92b4bcf-dc7b-448e-bf39-87daf7614ee4" (UID: "d92b4bcf-dc7b-448e-bf39-87daf7614ee4"). InnerVolumeSpecName "kube-api-access-s5757". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:30.089329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:30.089298 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5757\" (UniqueName: \"kubernetes.io/projected/d92b4bcf-dc7b-448e-bf39-87daf7614ee4-kube-api-access-s5757\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.901374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:30.901339 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr" Oct 30 02:01:30.951301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:30.951272 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr"] Oct 30 02:01:30.959980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:30.959950 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-547c679c5-t8nvr"] Oct 30 02:01:31.509178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.503455 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z"] Oct 30 02:01:31.509178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.504045 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" podUID="3ebfb1cb-be63-4866-98f7-81d3f488ce91" containerName="maas-api" containerID="cri-o://76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912" gracePeriod=30 Oct 30 02:01:31.662580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.662543 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g"] Oct 30 02:01:31.663075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.662770 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" podUID="162eafa6-0246-407f-8a94-8a152e65a4a8" containerName="payload-processing" containerID="cri-o://2c79e41286d319d7095d0963f4527f629767913771b91efe0c7ffeee08209385" gracePeriod=30 Oct 30 02:01:31.751931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.751481 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92b4bcf-dc7b-448e-bf39-87daf7614ee4" path="/var/lib/kubelet/pods/d92b4bcf-dc7b-448e-bf39-87daf7614ee4/volumes" Oct 30 02:01:31.829955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.829564 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:01:31.910478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.910441 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 02:01:31.940723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.940693 6275 generic.go:356] "Generic (PLEG): container finished" podID="162eafa6-0246-407f-8a94-8a152e65a4a8" containerID="2c79e41286d319d7095d0963f4527f629767913771b91efe0c7ffeee08209385" exitCode=0 Oct 30 02:01:31.941003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.940985 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" event={"ID":"162eafa6-0246-407f-8a94-8a152e65a4a8","Type":"ContainerDied","Data":"2c79e41286d319d7095d0963f4527f629767913771b91efe0c7ffeee08209385"} Oct 30 02:01:31.942964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.942933 6275 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb1cb-be63-4866-98f7-81d3f488ce91" containerID="76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912" exitCode=0 Oct 30 02:01:31.943327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.943303 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" event={"ID":"3ebfb1cb-be63-4866-98f7-81d3f488ce91","Type":"ContainerDied","Data":"76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912"} Oct 30 02:01:31.943505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.943486 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" event={"ID":"3ebfb1cb-be63-4866-98f7-81d3f488ce91","Type":"ContainerDied","Data":"eae3804dff0e69be6280093a1b20aa006a0b154a9b0d79e7b46a22e759e7174a"} Oct 30 02:01:31.943625 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.943612 6275 scope.go:122] "RemoveContainer" containerID="76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912" Oct 30 02:01:31.944003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.943864 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z" Oct 30 02:01:31.958843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.958820 6275 scope.go:122] "RemoveContainer" containerID="76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912" Oct 30 02:01:31.959370 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:31.959346 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912\": container with ID starting with 76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912 not found: ID does not exist" containerID="76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912" Oct 30 02:01:31.959520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:31.959496 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912"} err="failed to get container status \"76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912\": rpc error: code = NotFound desc = could not find container \"76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912\": container with ID starting with 76e110c3c203ce9e2541237af08b634b25ca3712fbc302880c708ba0cc26c912 not found: ID does not exist" Oct 30 02:01:32.036734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.036700 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca\") pod \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " Oct 30 02:01:32.036933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.036782 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk\" (UniqueName: \"kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk\") pod \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " Oct 30 02:01:32.037005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.036959 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs\") pod \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " Oct 30 02:01:32.037005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.036989 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls\") pod \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\" (UID: \"3ebfb1cb-be63-4866-98f7-81d3f488ce91\") " Oct 30 02:01:32.037540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.037509 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca" pod "3ebfb1cb-be63-4866-98f7-81d3f488ce91" (UID: "3ebfb1cb-be63-4866-98f7-81d3f488ce91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:32.037886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.037864 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfb1cb-be63-4866-98f7-81d3f488ce91-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.039785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.039761 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs" pod "3ebfb1cb-be63-4866-98f7-81d3f488ce91" (UID: "3ebfb1cb-be63-4866-98f7-81d3f488ce91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:32.039906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.039788 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls" pod "3ebfb1cb-be63-4866-98f7-81d3f488ce91" (UID: "3ebfb1cb-be63-4866-98f7-81d3f488ce91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:32.040857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.040833 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk" pod "3ebfb1cb-be63-4866-98f7-81d3f488ce91" (UID: "3ebfb1cb-be63-4866-98f7-81d3f488ce91"). InnerVolumeSpecName "kube-api-access-h6xkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:32.139200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.136253 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 02:01:32.189154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.183386 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf\" (UniqueName: \"kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf\") pod \"162eafa6-0246-407f-8a94-8a152e65a4a8\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " Oct 30 02:01:32.189154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.183455 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume\") pod \"162eafa6-0246-407f-8a94-8a152e65a4a8\" (UID: \"162eafa6-0246-407f-8a94-8a152e65a4a8\") " Oct 30 02:01:32.189154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.184059 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xkk\" (UniqueName: \"kubernetes.io/projected/3ebfb1cb-be63-4866-98f7-81d3f488ce91-kube-api-access-h6xkk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.189154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.184085 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.189154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.184100 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebfb1cb-be63-4866-98f7-81d3f488ce91-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.249325 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.249163 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf" pod "162eafa6-0246-407f-8a94-8a152e65a4a8" (UID: "162eafa6-0246-407f-8a94-8a152e65a4a8"). InnerVolumeSpecName "kube-api-access-htrxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:32.314545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.270264 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume" pod "162eafa6-0246-407f-8a94-8a152e65a4a8" (UID: "162eafa6-0246-407f-8a94-8a152e65a4a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:32.314545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.270568 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" podUID="6abcc06a-07aa-430c-81a5-31ce5b4e916f" containerName="payload-processing" containerID="cri-o://2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa" gracePeriod=25 Oct 30 02:01:32.314545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.285924 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrxf\" (UniqueName: \"kubernetes.io/projected/162eafa6-0246-407f-8a94-8a152e65a4a8-kube-api-access-htrxf\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.314545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.285953 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162eafa6-0246-407f-8a94-8a152e65a4a8-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.377243 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.377210 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z"] Oct 30 02:01:32.385282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.385245 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7542fa-6588fc5b56-fgn9z"] Oct 30 02:01:32.630113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.630086 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:01:32.692362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.692283 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume\") pod \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " Oct 30 02:01:32.692501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.692431 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5\" (UniqueName: \"kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5\") pod \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\" (UID: \"6abcc06a-07aa-430c-81a5-31ce5b4e916f\") " Oct 30 02:01:32.692759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.692727 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume" pod "6abcc06a-07aa-430c-81a5-31ce5b4e916f" (UID: "6abcc06a-07aa-430c-81a5-31ce5b4e916f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:32.693173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.693105 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abcc06a-07aa-430c-81a5-31ce5b4e916f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.694902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.694867 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5" pod "6abcc06a-07aa-430c-81a5-31ce5b4e916f" (UID: "6abcc06a-07aa-430c-81a5-31ce5b4e916f"). InnerVolumeSpecName "kube-api-access-2j6s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:32.794671 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.794639 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6s5\" (UniqueName: \"kubernetes.io/projected/6abcc06a-07aa-430c-81a5-31ce5b4e916f-kube-api-access-2j6s5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.952105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.952025 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" event={"ID":"162eafa6-0246-407f-8a94-8a152e65a4a8","Type":"ContainerDied","Data":"4646fb66aae547e64aef2283f806959299d5dd1d2b50bfb16666cd133fb89e21"} Oct 30 02:01:32.952105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.952096 6275 scope.go:122] "RemoveContainer" containerID="2c79e41286d319d7095d0963f4527f629767913771b91efe0c7ffeee08209385" Oct 30 02:01:32.952564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.952258 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g" Oct 30 02:01:32.962866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.961865 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" Oct 30 02:01:32.962866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.961815 6275 generic.go:356] "Generic (PLEG): container finished" podID="6abcc06a-07aa-430c-81a5-31ce5b4e916f" containerID="2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa" exitCode=0 Oct 30 02:01:32.962866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.961867 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" event={"ID":"6abcc06a-07aa-430c-81a5-31ce5b4e916f","Type":"ContainerDied","Data":"2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa"} Oct 30 02:01:32.962866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.961977 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn" event={"ID":"6abcc06a-07aa-430c-81a5-31ce5b4e916f","Type":"ContainerDied","Data":"c6e33e06fee41ab4197825c8d59fa2cd77931c4d32c094fdf7db31dd5bcdf3f9"} Oct 30 02:01:32.981881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.981813 6275 scope.go:122] "RemoveContainer" containerID="2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa" Oct 30 02:01:32.994024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.993992 6275 scope.go:122] "RemoveContainer" containerID="2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa" Oct 30 02:01:32.994355 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:32.994334 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa\": container with ID starting with 2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa not found: ID does not exist" containerID="2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa" Oct 30 02:01:32.994431 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:32.994366 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa"} err="failed to get container status \"2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa\": rpc error: code = NotFound desc = could not find container \"2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa\": container with ID starting with 2c929975e2fd694e6e1d4c072ee0f84c61e11cecd3f7ee600d72ff703988ddfa not found: ID does not exist" Oct 30 02:01:33.019729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.019697 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g"] Oct 30 02:01:33.026887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.026859 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7542fa-5fff779d4-p5p5g"] Oct 30 02:01:33.057109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.057073 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn"] Oct 30 02:01:33.130370 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.130334 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-6bcf449fcd-wd9mn"] Oct 30 02:01:33.740950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.740875 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162eafa6-0246-407f-8a94-8a152e65a4a8" path="/var/lib/kubelet/pods/162eafa6-0246-407f-8a94-8a152e65a4a8/volumes" Oct 30 02:01:33.741427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.741410 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ebfb1cb-be63-4866-98f7-81d3f488ce91" path="/var/lib/kubelet/pods/3ebfb1cb-be63-4866-98f7-81d3f488ce91/volumes" Oct 30 02:01:33.741791 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:33.741778 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abcc06a-07aa-430c-81a5-31ce5b4e916f" path="/var/lib/kubelet/pods/6abcc06a-07aa-430c-81a5-31ce5b4e916f/volumes" Oct 30 02:01:35.193715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:35.193680 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647"] Oct 30 02:01:35.194220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:35.193948 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" containerID="cri-o://39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369" gracePeriod=30 Oct 30 02:01:38.167980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:38.167941 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:01:38.168480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:38.168003 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:01:40.472069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.472037 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr"] Oct 30 02:01:40.473009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.472992 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d92b4bcf-dc7b-448e-bf39-87daf7614ee4" containerName="payload-pre-processing" Oct 30 02:01:40.473068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473012 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92b4bcf-dc7b-448e-bf39-87daf7614ee4" containerName="payload-pre-processing" Oct 30 02:01:40.473068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473033 6275 cpu_manager.go:414] "containerMap: removing container" podUID="3ebfb1cb-be63-4866-98f7-81d3f488ce91" containerName="maas-api" Oct 30 02:01:40.473068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473041 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ebfb1cb-be63-4866-98f7-81d3f488ce91" containerName="maas-api" Oct 30 02:01:40.473068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473062 6275 cpu_manager.go:414] "containerMap: removing container" podUID="92167404-cffd-4c62-a94e-d0e48795ccad" containerName="maas-api" Oct 30 02:01:40.473224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473070 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92167404-cffd-4c62-a94e-d0e48795ccad" containerName="maas-api" Oct 30 02:01:40.473224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473086 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6abcc06a-07aa-430c-81a5-31ce5b4e916f" containerName="payload-processing" Oct 30 02:01:40.473224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473094 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abcc06a-07aa-430c-81a5-31ce5b4e916f" containerName="payload-processing" Oct 30 02:01:40.473224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473160 6275 cpu_manager.go:414] "containerMap: removing container" podUID="162eafa6-0246-407f-8a94-8a152e65a4a8" containerName="payload-processing" Oct 30 02:01:40.473224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473169 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162eafa6-0246-407f-8a94-8a152e65a4a8" containerName="payload-processing" Oct 30 02:01:40.473377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473296 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abcc06a-07aa-430c-81a5-31ce5b4e916f" containerName="payload-processing" Oct 30 02:01:40.473377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473317 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="162eafa6-0246-407f-8a94-8a152e65a4a8" containerName="payload-processing" Oct 30 02:01:40.473377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473331 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92b4bcf-dc7b-448e-bf39-87daf7614ee4" containerName="payload-pre-processing" Oct 30 02:01:40.473377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473343 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ebfb1cb-be63-4866-98f7-81d3f488ce91" containerName="maas-api" Oct 30 02:01:40.473377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.473354 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="92167404-cffd-4c62-a94e-d0e48795ccad" containerName="maas-api" Oct 30 02:01:40.477564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.477540 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.495007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.494939 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 02:01:40.509649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.506597 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-95daa2-serving-cert\"" Oct 30 02:01:40.521183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.516976 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr"] Oct 30 02:01:40.526704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.526672 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.526830 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.526815 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.526880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.526866 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.526918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.526908 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.527053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527038 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.527117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527104 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.527210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527194 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.527265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527245 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.527316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527303 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data\") pod \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\" (UID: \"6d3afe93-cb6b-473e-85d5-4fae2ed27555\") " Oct 30 02:01:40.527689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527662 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.527806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527761 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.527888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527831 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdblk\" (UniqueName: \"kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.527888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527882 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.527992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.527919 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:40.528086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.528062 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:40.528343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.528313 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:40.528475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.528360 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:40.528475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.528410 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.528475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.528429 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.528475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.528443 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:40.531389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.531351 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:40.531567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.531533 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:40.531671 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.531624 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:40.531856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.531816 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk" pod "6d3afe93-cb6b-473e-85d5-4fae2ed27555" (UID: "6d3afe93-cb6b-473e-85d5-4fae2ed27555"). InnerVolumeSpecName "kube-api-access-qjnvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:40.629654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.629618 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.629850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.629710 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.629850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.629773 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdblk\" (UniqueName: \"kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.629850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.629815 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.630071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630032 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630058 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjnvk\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-kube-api-access-qjnvk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630075 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630090 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630104 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3afe93-cb6b-473e-85d5-4fae2ed27555-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630132 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630146 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3afe93-cb6b-473e-85d5-4fae2ed27555-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:40.630955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.630931 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.632833 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:40.632807 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-serving-cert: secret "maas-api-e2e-praxis-a-95daa2-serving-cert" not found Oct 30 02:01:40.633049 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:40.633035 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls podName:ae01f56c-6259-4e01-a320-e5978662deb3 nodeName:}" failed. No retries permitted until 2026-10-30 02:01:41.133012205 +0000 UTC m=+1929.524819831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls") pod "maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" (UID: "ae01f56c-6259-4e01-a320-e5978662deb3") : secret "maas-api-e2e-praxis-a-95daa2-serving-cert" not found Oct 30 02:01:40.635942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.635890 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:40.722334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:40.720342 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdblk\" (UniqueName: \"kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:41.048649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.048617 6275 generic.go:356] "Generic (PLEG): container finished" podID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerID="39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369" exitCode=0 Oct 30 02:01:41.048857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.048675 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" Oct 30 02:01:41.048857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.048699 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" event={"ID":"6d3afe93-cb6b-473e-85d5-4fae2ed27555","Type":"ContainerDied","Data":"39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369"} Oct 30 02:01:41.048857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.048760 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647" event={"ID":"6d3afe93-cb6b-473e-85d5-4fae2ed27555","Type":"ContainerDied","Data":"2330916e8e9c8c329629ca7bca9bb747b5730be3d30e5236b7f92310f17694ec"} Oct 30 02:01:41.048857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.048791 6275 scope.go:122] "RemoveContainer" containerID="39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369" Oct 30 02:01:41.061091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.061067 6275 scope.go:122] "RemoveContainer" containerID="be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d" Oct 30 02:01:41.071236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.071216 6275 scope.go:122] "RemoveContainer" containerID="39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369" Oct 30 02:01:41.071518 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:41.071491 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369\": container with ID starting with 39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369 not found: ID does not exist" containerID="39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369" Oct 30 02:01:41.071575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.071530 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369"} err="failed to get container status \"39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369\": rpc error: code = NotFound desc = could not find container \"39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369\": container with ID starting with 39815f3fd3682cca617e612f9a7320f17266dc264cb63503208d2db271386369 not found: ID does not exist" Oct 30 02:01:41.071575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.071555 6275 scope.go:122] "RemoveContainer" containerID="be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d" Oct 30 02:01:41.071801 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:41.071783 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d\": container with ID starting with be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d not found: ID does not exist" containerID="be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d" Oct 30 02:01:41.071865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.071811 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d"} err="failed to get container status \"be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d\": rpc error: code = NotFound desc = could not find container \"be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d\": container with ID starting with be365ca3b9db28def68b6653b58cdf328976c420950d6819f6eb71dfb0a93d3d not found: ID does not exist" Oct 30 02:01:41.096714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.096690 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647"] Oct 30 02:01:41.100443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.100423 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7542fa-openshift-default-65c497d8b9-h9647"] Oct 30 02:01:41.145005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.144965 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:41.147436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.147415 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:41.415275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.415167 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:41.741927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.741902 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" path="/var/lib/kubelet/pods/6d3afe93-cb6b-473e-85d5-4fae2ed27555/volumes" Oct 30 02:01:41.742418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:41.742398 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr"] Oct 30 02:01:41.743985 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:41.743955 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae01f56c_6259_4e01_a320_e5978662deb3.slice/crio-32861d74b79b6c6e2b868828d47e37cca6cef64aa5b59eebee7253c6f0b42bb1 WatchSource:0}: Error finding container 32861d74b79b6c6e2b868828d47e37cca6cef64aa5b59eebee7253c6f0b42bb1: Status 404 returned error can't find the container with id 32861d74b79b6c6e2b868828d47e37cca6cef64aa5b59eebee7253c6f0b42bb1 Oct 30 02:01:42.058749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:42.058711 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" event={"ID":"ae01f56c-6259-4e01-a320-e5978662deb3","Type":"ContainerStarted","Data":"32861d74b79b6c6e2b868828d47e37cca6cef64aa5b59eebee7253c6f0b42bb1"} Oct 30 02:01:43.076406 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:43.076369 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" event={"ID":"ae01f56c-6259-4e01-a320-e5978662deb3","Type":"ContainerStarted","Data":"2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2"} Oct 30 02:01:43.076944 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:43.076596 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:43.116094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:43.116052 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" podStartSLOduration=2.5697204620000003 podStartE2EDuration="3.116040432s" totalImagesPullingTime="546.31997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:43.114969529 +0000 UTC m=+1931.506777159" watchObservedRunningTime="2026-10-30 02:01:43.116040432 +0000 UTC m=+1931.507848064" Oct 30 02:01:46.696735 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:46.696695 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:01:47.248142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.246017 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j"] Oct 30 02:01:47.248142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.247074 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" Oct 30 02:01:47.248142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.247095 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" Oct 30 02:01:47.248142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.247304 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" Oct 30 02:01:47.263007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.261584 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.278500 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.278111 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-145336\"" Oct 30 02:01:47.279789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.279757 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j"] Oct 30 02:01:47.295523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.295416 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-145336-dockercfg-hxz62\"" Oct 30 02:01:47.401081 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.401045 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742"] Oct 30 02:01:47.401987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.401967 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" Oct 30 02:01:47.402060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.401989 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" Oct 30 02:01:47.402155 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.402139 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3afe93-cb6b-473e-85d5-4fae2ed27555" containerName="istio-proxy" Oct 30 02:01:47.405692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.405673 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.428182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.428153 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742"] Oct 30 02:01:47.451722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.451682 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.451890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.451846 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffp2s\" (UniqueName: \"kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s\") pod \"payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.554712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.554671 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.554890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.554782 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59s5\" (UniqueName: \"kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5\") pod \"payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.554890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.554819 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffp2s\" (UniqueName: \"kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s\") pod \"payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.555099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.555083 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.555410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.555390 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.576762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.576729 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffp2s\" (UniqueName: \"kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s\") pod \"payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.604603 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.604573 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:47.656090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.656033 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59s5\" (UniqueName: \"kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5\") pod \"payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.656341 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.656273 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.657025 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.656996 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.685537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.685433 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59s5\" (UniqueName: \"kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5\") pod \"payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.720343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.720306 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:47.919378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:47.919349 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j"] Oct 30 02:01:47.921549 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:47.921519 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811316d1_343e_44b8_86f9_2a100937d91d.slice/crio-354403d92585362cafb0ca37defc768202cb26b423a616be66a9f82767da4d6a WatchSource:0}: Error finding container 354403d92585362cafb0ca37defc768202cb26b423a616be66a9f82767da4d6a: Status 404 returned error can't find the container with id 354403d92585362cafb0ca37defc768202cb26b423a616be66a9f82767da4d6a Oct 30 02:01:48.066813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:48.066783 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742"] Oct 30 02:01:48.068494 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:48.068468 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9022ffe1_ea98_4fc1_b4ad_344020ff4a3f.slice/crio-180ca05b746e5cd173fe47d4ca490f9d1c5e194acdf27e1b03b173b9dc9095a4 WatchSource:0}: Error finding container 180ca05b746e5cd173fe47d4ca490f9d1c5e194acdf27e1b03b173b9dc9095a4: Status 404 returned error can't find the container with id 180ca05b746e5cd173fe47d4ca490f9d1c5e194acdf27e1b03b173b9dc9095a4 Oct 30 02:01:48.116595 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:48.116565 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" event={"ID":"811316d1-343e-44b8-86f9-2a100937d91d","Type":"ContainerStarted","Data":"1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c"} Oct 30 02:01:48.116730 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:48.116623 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" event={"ID":"811316d1-343e-44b8-86f9-2a100937d91d","Type":"ContainerStarted","Data":"354403d92585362cafb0ca37defc768202cb26b423a616be66a9f82767da4d6a"} Oct 30 02:01:48.116730 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:48.116656 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:01:48.117966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:48.117940 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" event={"ID":"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f","Type":"ContainerStarted","Data":"180ca05b746e5cd173fe47d4ca490f9d1c5e194acdf27e1b03b173b9dc9095a4"} Oct 30 02:01:49.088288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.088259 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:01:49.125927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.125892 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" event={"ID":"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f","Type":"ContainerStarted","Data":"a156b9b321899a75bca5e3326313c3785ced26159f4fb37c581ccba5d4a641fe"} Oct 30 02:01:49.126094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.125958 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:01:49.130241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.130196 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" podStartSLOduration=2.130182152 podStartE2EDuration="2.130182152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:48.160397803 +0000 UTC m=+1936.552205436" watchObservedRunningTime="2026-10-30 02:01:49.130182152 +0000 UTC m=+1937.521989785" Oct 30 02:01:49.190030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.189965 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" podStartSLOduration=2.189946655 podStartE2EDuration="2.189946655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:49.186505288 +0000 UTC m=+1937.578312922" watchObservedRunningTime="2026-10-30 02:01:49.189946655 +0000 UTC m=+1937.581754289" Oct 30 02:01:49.573211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.563542 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp"] Oct 30 02:01:49.577005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.576975 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.597579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.597548 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38bc0d-dockercfg-pwjdd\"" Oct 30 02:01:49.598611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.598586 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38bc0d\"" Oct 30 02:01:49.610273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.608891 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp"] Oct 30 02:01:49.647336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.647295 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.708676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.708530 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6cm\" (UniqueName: \"kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.788267 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.788232 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf"] Oct 30 02:01:49.794065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.794029 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf"] Oct 30 02:01:49.794222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.794160 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:49.811017 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.810983 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62sh8\" (UniqueName: \"kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8\") pod \"payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:49.811217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.811040 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:49.811217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.811166 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6cm\" (UniqueName: \"kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.811217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.811216 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.811949 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.811925 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.832991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.832932 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6cm\" (UniqueName: \"kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:49.912691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.912658 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62sh8\" (UniqueName: \"kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8\") pod \"payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:49.912853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.912708 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:49.913449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.913428 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:49.934625 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:49.934595 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62sh8\" (UniqueName: \"kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8\") pod \"payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:50.013325 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:50.013289 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:50.113045 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:50.112956 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:50.371908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:50.371811 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp"] Oct 30 02:01:50.378046 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:50.377991 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76274492_bf77_417d_9b88_59ca6f344e48.slice/crio-76871a6fc62303ab1450baf95709e5aca9f8257b4f2d10ecd365f8b2db60752a WatchSource:0}: Error finding container 76871a6fc62303ab1450baf95709e5aca9f8257b4f2d10ecd365f8b2db60752a: Status 404 returned error can't find the container with id 76871a6fc62303ab1450baf95709e5aca9f8257b4f2d10ecd365f8b2db60752a Oct 30 02:01:50.450657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:50.450634 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf"] Oct 30 02:01:50.456554 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:01:50.456527 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586b35e3_7fb1_4846_a055_126f5c58f30f.slice/crio-4991781aed494379cb19e79c66cebff4035b412f8b5d5abcf01f28ac075b2370 WatchSource:0}: Error finding container 4991781aed494379cb19e79c66cebff4035b412f8b5d5abcf01f28ac075b2370: Status 404 returned error can't find the container with id 4991781aed494379cb19e79c66cebff4035b412f8b5d5abcf01f28ac075b2370 Oct 30 02:01:51.145102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.145066 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" event={"ID":"586b35e3-7fb1-4846-a055-126f5c58f30f","Type":"ContainerStarted","Data":"c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c"} Oct 30 02:01:51.145565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.145156 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" event={"ID":"586b35e3-7fb1-4846-a055-126f5c58f30f","Type":"ContainerStarted","Data":"4991781aed494379cb19e79c66cebff4035b412f8b5d5abcf01f28ac075b2370"} Oct 30 02:01:51.145565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.145219 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:01:51.146963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.146929 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" event={"ID":"76274492-bf77-417d-9b88-59ca6f344e48","Type":"ContainerStarted","Data":"e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b"} Oct 30 02:01:51.147076 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.146968 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" event={"ID":"76274492-bf77-417d-9b88-59ca6f344e48","Type":"ContainerStarted","Data":"76871a6fc62303ab1450baf95709e5aca9f8257b4f2d10ecd365f8b2db60752a"} Oct 30 02:01:51.147076 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.146997 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:01:51.186601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.186554 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" podStartSLOduration=2.18654044 podStartE2EDuration="2.18654044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:51.183432435 +0000 UTC m=+1939.575240069" watchObservedRunningTime="2026-10-30 02:01:51.18654044 +0000 UTC m=+1939.578348073" Oct 30 02:01:51.223089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:51.223042 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" podStartSLOduration=2.223030742 podStartE2EDuration="2.223030742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:51.221207201 +0000 UTC m=+1939.613014827" watchObservedRunningTime="2026-10-30 02:01:51.223030742 +0000 UTC m=+1939.614838375" Oct 30 02:01:53.687165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:53.687101 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf"] Oct 30 02:01:53.687779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:53.687431 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" podUID="f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" containerName="maas-api" containerID="cri-o://c260cd67eadd5caf8dd565f19ba26e407d59145cfc16078c0963085f9960b6fd" gracePeriod=30 Oct 30 02:01:54.183281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.182978 6275 generic.go:356] "Generic (PLEG): container finished" podID="f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" containerID="c260cd67eadd5caf8dd565f19ba26e407d59145cfc16078c0963085f9960b6fd" exitCode=0 Oct 30 02:01:54.183281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.183172 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" event={"ID":"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94","Type":"ContainerDied","Data":"c260cd67eadd5caf8dd565f19ba26e407d59145cfc16078c0963085f9960b6fd"} Oct 30 02:01:54.183281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.183225 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" event={"ID":"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94","Type":"ContainerDied","Data":"5828a4eb3a7120f9a4192b20824c2468890350c4221b7fdc3492bbd3b2029f41"} Oct 30 02:01:54.183281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.183248 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5828a4eb3a7120f9a4192b20824c2468890350c4221b7fdc3492bbd3b2029f41" Oct 30 02:01:54.222143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.218336 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:01:54.349956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.349923 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca\") pod \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " Oct 30 02:01:54.350202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.349982 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd\" (UniqueName: \"kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd\") pod \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " Oct 30 02:01:54.350202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.350024 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls\") pod \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " Oct 30 02:01:54.350202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.350114 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs\") pod \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\" (UID: \"f0bda3d0-68bb-4aab-958d-4b9f55dbdb94\") " Oct 30 02:01:54.350444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.350415 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca" pod "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" (UID: "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:54.350809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.350653 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:54.353277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.353244 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd" pod "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" (UID: "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94"). InnerVolumeSpecName "kube-api-access-sqnkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:54.354186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.354161 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls" pod "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" (UID: "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:54.354783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.354756 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs" pod "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" (UID: "f0bda3d0-68bb-4aab-958d-4b9f55dbdb94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:54.451475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.451439 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqnkd\" (UniqueName: \"kubernetes.io/projected/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-kube-api-access-sqnkd\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:54.451475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.451469 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:54.451475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:54.451479 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:55.219146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:55.218580 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf" Oct 30 02:01:55.369235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:55.369193 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf"] Oct 30 02:01:55.381220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:55.381178 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-356082-597b97fb45-v4btf"] Oct 30 02:01:55.740915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:55.740883 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" path="/var/lib/kubelet/pods/f0bda3d0-68bb-4aab-958d-4b9f55dbdb94/volumes" Oct 30 02:01:56.410421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:56.410390 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph"] Oct 30 02:01:56.410819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:56.410619 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" podUID="98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" containerName="payload-processing" containerID="cri-o://4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b" gracePeriod=30 Oct 30 02:01:56.425193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:56.425162 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v"] Oct 30 02:01:56.425447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:56.425400 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" podUID="b554eb84-c181-48a2-aae5-7dfaab59ab36" containerName="payload-pre-processing" containerID="cri-o://7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c" gracePeriod=30 Oct 30 02:01:56.974066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:56.974040 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:01:56.978972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:56.978949 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:01:57.111196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.111104 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd\" (UniqueName: \"kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd\") pod \"b554eb84-c181-48a2-aae5-7dfaab59ab36\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " Oct 30 02:01:57.111409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.111314 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume\") pod \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " Oct 30 02:01:57.111610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.111574 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume\") pod \"b554eb84-c181-48a2-aae5-7dfaab59ab36\" (UID: \"b554eb84-c181-48a2-aae5-7dfaab59ab36\") " Oct 30 02:01:57.111740 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.111710 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv\" (UniqueName: \"kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv\") pod \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\" (UID: \"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d\") " Oct 30 02:01:57.111809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.111749 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume" pod "98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" (UID: "98ab53a6-7d7f-4416-9aa6-bcd7cc32052d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:57.112618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.112046 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume" pod "b554eb84-c181-48a2-aae5-7dfaab59ab36" (UID: "b554eb84-c181-48a2-aae5-7dfaab59ab36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:57.112618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.112471 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:57.112618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.112489 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b554eb84-c181-48a2-aae5-7dfaab59ab36-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:57.113803 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.113782 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd" pod "b554eb84-c181-48a2-aae5-7dfaab59ab36" (UID: "b554eb84-c181-48a2-aae5-7dfaab59ab36"). InnerVolumeSpecName "kube-api-access-lvdpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:57.115210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.115191 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv" pod "98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" (UID: "98ab53a6-7d7f-4416-9aa6-bcd7cc32052d"). InnerVolumeSpecName "kube-api-access-krgqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:57.213522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.213436 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgqv\" (UniqueName: \"kubernetes.io/projected/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d-kube-api-access-krgqv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:57.213522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.213464 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdpd\" (UniqueName: \"kubernetes.io/projected/b554eb84-c181-48a2-aae5-7dfaab59ab36-kube-api-access-lvdpd\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:01:57.234272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.234232 6275 generic.go:356] "Generic (PLEG): container finished" podID="98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" containerID="4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b" exitCode=0 Oct 30 02:01:57.234455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.234291 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" Oct 30 02:01:57.234455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.234317 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" event={"ID":"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d","Type":"ContainerDied","Data":"4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b"} Oct 30 02:01:57.234455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.234377 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph" event={"ID":"98ab53a6-7d7f-4416-9aa6-bcd7cc32052d","Type":"ContainerDied","Data":"e674a4a6ae8af0391a0bbc4b8287858d2dc4176673591e4112397c2c397ddc1d"} Oct 30 02:01:57.234455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.234407 6275 scope.go:122] "RemoveContainer" containerID="4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b" Oct 30 02:01:57.235727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.235706 6275 generic.go:356] "Generic (PLEG): container finished" podID="b554eb84-c181-48a2-aae5-7dfaab59ab36" containerID="7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c" exitCode=0 Oct 30 02:01:57.235839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.235741 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" event={"ID":"b554eb84-c181-48a2-aae5-7dfaab59ab36","Type":"ContainerDied","Data":"7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c"} Oct 30 02:01:57.235839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.235798 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" event={"ID":"b554eb84-c181-48a2-aae5-7dfaab59ab36","Type":"ContainerDied","Data":"39da7ccfce8c774587fc0839105c822729f52b563ab7dea53d5dd03d557cbc23"} Oct 30 02:01:57.235839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.235758 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v" Oct 30 02:01:57.245604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.245588 6275 scope.go:122] "RemoveContainer" containerID="4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b" Oct 30 02:01:57.246059 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:57.245932 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b\": container with ID starting with 4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b not found: ID does not exist" containerID="4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b" Oct 30 02:01:57.246059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.245969 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b"} err="failed to get container status \"4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b\": rpc error: code = NotFound desc = could not find container \"4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b\": container with ID starting with 4875f2bed6bac12a7d32c1390ddad9843d9aaefb7ed28905a1f5ac828f880f9b not found: ID does not exist" Oct 30 02:01:57.246059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.245988 6275 scope.go:122] "RemoveContainer" containerID="7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c" Oct 30 02:01:57.257526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.257506 6275 scope.go:122] "RemoveContainer" containerID="7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c" Oct 30 02:01:57.289163 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:01:57.278528 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c\": container with ID starting with 7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c not found: ID does not exist" containerID="7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c" Oct 30 02:01:57.289163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.278581 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c"} err="failed to get container status \"7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c\": rpc error: code = NotFound desc = could not find container \"7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c\": container with ID starting with 7edbcd87fa3644b19d8c57536ffa215f0d49fae6bc562905df7caf40a711826c not found: ID does not exist" Oct 30 02:01:57.342026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.341993 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v"] Oct 30 02:01:57.345351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.345298 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-5ddcc994b7-q6l9v"] Oct 30 02:01:57.360543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.360516 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph"] Oct 30 02:01:57.367212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.367183 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-dd777b898-2fzph"] Oct 30 02:01:57.741753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.741720 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" path="/var/lib/kubelet/pods/98ab53a6-7d7f-4416-9aa6-bcd7cc32052d/volumes" Oct 30 02:01:57.742218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:57.742201 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b554eb84-c181-48a2-aae5-7dfaab59ab36" path="/var/lib/kubelet/pods/b554eb84-c181-48a2-aae5-7dfaab59ab36/volumes" Oct 30 02:01:59.127774 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:01:59.127739 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:02:00.138180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.138145 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:02:00.912027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.911994 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr"] Oct 30 02:02:00.912716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.912697 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" containerName="maas-api" Oct 30 02:02:00.912716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.912716 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" containerName="maas-api" Oct 30 02:02:00.912951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.912733 6275 cpu_manager.go:414] "containerMap: removing container" podUID="98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" containerName="payload-processing" Oct 30 02:02:00.912951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.912739 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" containerName="payload-processing" Oct 30 02:02:00.912951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.912749 6275 cpu_manager.go:414] "containerMap: removing container" podUID="b554eb84-c181-48a2-aae5-7dfaab59ab36" containerName="payload-pre-processing" Oct 30 02:02:00.912951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.912757 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b554eb84-c181-48a2-aae5-7dfaab59ab36" containerName="payload-pre-processing" Oct 30 02:02:00.913488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.913469 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bda3d0-68bb-4aab-958d-4b9f55dbdb94" containerName="maas-api" Oct 30 02:02:00.913548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.913497 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ab53a6-7d7f-4416-9aa6-bcd7cc32052d" containerName="payload-processing" Oct 30 02:02:00.913548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.913508 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b554eb84-c181-48a2-aae5-7dfaab59ab36" containerName="payload-pre-processing" Oct 30 02:02:00.919225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.919199 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.936328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.936297 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-168989-openshift-default-dockercfg-mxs57\"" Oct 30 02:02:00.950661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.950628 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr"] Oct 30 02:02:00.988482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988448 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.988666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988511 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.988666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988578 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.988666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988617 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.988827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988723 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfpr8\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.988890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988866 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.988948 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988919 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.989006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.988969 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:00.989113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:00.989099 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090564 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090600 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090625 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090679 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfpr8\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090751 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090774 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.090815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090790 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090910 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.090986 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.091272 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.091415 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.091469 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.091462 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.091689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.091652 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.093643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.093623 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.093808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.093789 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.134658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.114224 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.134658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.134589 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfpr8\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8\") pod \"e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.258346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.258002 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:01.704009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.703981 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr"] Oct 30 02:02:01.707149 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:02:01.707105 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4979a4_b725_4497_bc66_6c6779fbcff1.slice/crio-7562e937ddd3deeb1143bb2a801c0f485a163cd087e948935e0fc47d1a6abba3 WatchSource:0}: Error finding container 7562e937ddd3deeb1143bb2a801c0f485a163cd087e948935e0fc47d1a6abba3: Status 404 returned error can't find the container with id 7562e937ddd3deeb1143bb2a801c0f485a163cd087e948935e0fc47d1a6abba3 Oct 30 02:02:01.709788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.709752 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:02:01.709872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.709825 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:02:01.709872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.709855 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:02:01.799642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.798702 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r"] Oct 30 02:02:01.803493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.800224 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" containerID="cri-o://0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1" gracePeriod=30 Oct 30 02:02:01.809734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.809706 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4"] Oct 30 02:02:01.809960 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:01.809937 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" containerID="cri-o://ff95ece34520d73b3a97f26ba32b14f3a52751feb8c265f1bb66211133aaeda2" gracePeriod=30 Oct 30 02:02:02.156580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:02.156362 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:02:02.156761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:02.156638 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:02:02.301488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:02.301447 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" event={"ID":"cf4979a4-b725-4497-bc66-6c6779fbcff1","Type":"ContainerStarted","Data":"398e81107bc9832c90de9435b6b8d0bf35c0859f9b58cfe1d2159a17d689edb2"} Oct 30 02:02:02.301981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:02.301515 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" event={"ID":"cf4979a4-b725-4497-bc66-6c6779fbcff1","Type":"ContainerStarted","Data":"7562e937ddd3deeb1143bb2a801c0f485a163cd087e948935e0fc47d1a6abba3"} Oct 30 02:02:02.345544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:02.345493 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podStartSLOduration=2.34547724 podStartE2EDuration="2.34547724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:02.34462817 +0000 UTC m=+1950.736435803" watchObservedRunningTime="2026-10-30 02:02:02.34547724 +0000 UTC m=+1950.737284873" Oct 30 02:02:03.259699 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:03.259661 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:03.265695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:03.265669 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:03.309966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:03.309935 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:03.310858 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:03.310836 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:02:04.689117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.689085 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5"] Oct 30 02:02:04.694065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.694045 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.716707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.715142 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5"] Oct 30 02:02:04.720270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.720242 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5c90f5\"/\"ipp-route-5c90f5-kserve-self-signed-certs\"" Oct 30 02:02:04.720761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.720727 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5c90f5\"/\"kube-root-ca.crt\"" Oct 30 02:02:04.720844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.720788 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5c90f5\"/\"openshift-service-ca.crt\"" Oct 30 02:02:04.726099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.726081 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5c90f5\"/\"default-dockercfg-42n74\"" Oct 30 02:02:04.818212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.818177 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.818372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.818311 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.818372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.818337 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.818495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.818382 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.818608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.818585 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6g5\" (UniqueName: \"kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.818642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.818633 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.920381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920328 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.920672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920425 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.920672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920572 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.920872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920819 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.921006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920959 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6g5\" (UniqueName: \"kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.921006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920992 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.921137 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.920828 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.921466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.921435 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.921565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.921488 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.923227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.923208 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.923632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.923610 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:04.950975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:04.950894 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6g5\" (UniqueName: \"kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5\") pod \"ipp-route-5c90f5-kserve-7568cdcc97-z48n5\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:05.040158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:05.024673 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:05.519358 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:02:05.519313 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod967bb5b5_0617_4eb8_8cd4_0543325c11a3.slice/crio-8d42f645f98f2b2460206650ed517c36d34893627747a8753cc8fde3eefe9a11 WatchSource:0}: Error finding container 8d42f645f98f2b2460206650ed517c36d34893627747a8753cc8fde3eefe9a11: Status 404 returned error can't find the container with id 8d42f645f98f2b2460206650ed517c36d34893627747a8753cc8fde3eefe9a11 Oct 30 02:02:05.521276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:05.521248 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5"] Oct 30 02:02:06.417658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:06.417616 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" event={"ID":"967bb5b5-0617-4eb8-8cd4-0543325c11a3","Type":"ContainerStarted","Data":"5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc"} Oct 30 02:02:06.418110 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:06.417671 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" event={"ID":"967bb5b5-0617-4eb8-8cd4-0543325c11a3","Type":"ContainerStarted","Data":"8d42f645f98f2b2460206650ed517c36d34893627747a8753cc8fde3eefe9a11"} Oct 30 02:02:06.466700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:06.466650 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" podStartSLOduration=2.466636651 podStartE2EDuration="2.466636651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:06.457105725 +0000 UTC m=+1954.848913359" watchObservedRunningTime="2026-10-30 02:02:06.466636651 +0000 UTC m=+1954.858444326" Oct 30 02:02:07.028418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.028373 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:07.088495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.088469 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 02:02:07.150301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150206 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150293 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150355 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150505 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150564 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150593 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150638 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.150816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150670 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150700 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150745 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert\") pod \"202f1f31-21e9-4309-be30-7807ac06e376\" (UID: \"202f1f31-21e9-4309-be30-7807ac06e376\") " Oct 30 02:02:07.150973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.150899 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.151587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.151115 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.151587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.151454 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:07.151587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.151540 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.151587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.151563 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.151587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.151578 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.151902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.151793 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.169154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.155638 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:07.169154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.155657 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.169154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.155667 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.169154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.165736 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m" pod "202f1f31-21e9-4309-be30-7807ac06e376" (UID: "202f1f31-21e9-4309-be30-7807ac06e376"). InnerVolumeSpecName "kube-api-access-qf47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.252922 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.252884 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.252922 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.252926 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.253157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.252940 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202f1f31-21e9-4309-be30-7807ac06e376-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.253157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.252954 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf47m\" (UniqueName: \"kubernetes.io/projected/202f1f31-21e9-4309-be30-7807ac06e376-kube-api-access-qf47m\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.253157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.252973 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202f1f31-21e9-4309-be30-7807ac06e376-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.253157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.252987 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202f1f31-21e9-4309-be30-7807ac06e376-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.431174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.431140 6275 generic.go:356] "Generic (PLEG): container finished" podID="a27dab51-2202-47b7-8081-28b6370de6e0" containerID="ff95ece34520d73b3a97f26ba32b14f3a52751feb8c265f1bb66211133aaeda2" exitCode=0 Oct 30 02:02:07.431859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.431385 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" event={"ID":"a27dab51-2202-47b7-8081-28b6370de6e0","Type":"ContainerDied","Data":"ff95ece34520d73b3a97f26ba32b14f3a52751feb8c265f1bb66211133aaeda2"} Oct 30 02:02:07.434189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.434164 6275 generic.go:356] "Generic (PLEG): container finished" podID="202f1f31-21e9-4309-be30-7807ac06e376" containerID="0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1" exitCode=0 Oct 30 02:02:07.434346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.434262 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" Oct 30 02:02:07.434346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.434290 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" event={"ID":"202f1f31-21e9-4309-be30-7807ac06e376","Type":"ContainerDied","Data":"0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1"} Oct 30 02:02:07.434346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.434331 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r" event={"ID":"202f1f31-21e9-4309-be30-7807ac06e376","Type":"ContainerDied","Data":"f03f25452e076244029525e7722402cebfd8aa5ce5061ba771811f3845b60659"} Oct 30 02:02:07.434505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.434359 6275 scope.go:122] "RemoveContainer" containerID="0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1" Oct 30 02:02:07.452179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.452156 6275 scope.go:122] "RemoveContainer" containerID="5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5" Oct 30 02:02:07.471568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.471546 6275 scope.go:122] "RemoveContainer" containerID="0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1" Oct 30 02:02:07.471895 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:07.471870 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1\": container with ID starting with 0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1 not found: ID does not exist" containerID="0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1" Oct 30 02:02:07.471996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.471903 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1"} err="failed to get container status \"0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1\": rpc error: code = NotFound desc = could not find container \"0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1\": container with ID starting with 0b629f496e05f5c2bccf7d573dae0cfc01344862412d980843ba705a8a378ce1 not found: ID does not exist" Oct 30 02:02:07.471996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.471922 6275 scope.go:122] "RemoveContainer" containerID="5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5" Oct 30 02:02:07.472272 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:07.472248 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5\": container with ID starting with 5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5 not found: ID does not exist" containerID="5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5" Oct 30 02:02:07.472347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.472280 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5"} err="failed to get container status \"5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5\": rpc error: code = NotFound desc = could not find container \"5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5\": container with ID starting with 5d2f0fec069e57e7bd86204f91eff96fbc6845c2d8aa2c9fc5775142c32d60b5 not found: ID does not exist" Oct 30 02:02:07.496356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.496327 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r"] Oct 30 02:02:07.507151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.507114 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rpv8r"] Oct 30 02:02:07.517016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.516993 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:02:07.556341 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556305 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556398 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556435 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556469 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556492 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556755 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556529 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556755 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556569 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556755 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556587 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.556931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556816 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.556931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556843 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.556931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556868 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy\") pod \"a27dab51-2202-47b7-8081-28b6370de6e0\" (UID: \"a27dab51-2202-47b7-8081-28b6370de6e0\") " Oct 30 02:02:07.557101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.556959 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.557101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557044 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.557101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557066 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:07.557542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557516 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.557641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557545 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27dab51-2202-47b7-8081-28b6370de6e0-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.557641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557560 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.557641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557570 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.557641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.557579 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.559339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.559317 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:07.559339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.559332 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "kube-api-access-ltzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.559495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.559447 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.559495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.559457 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy" pod "a27dab51-2202-47b7-8081-28b6370de6e0" (UID: "a27dab51-2202-47b7-8081-28b6370de6e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.658752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.658675 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzcd\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-kube-api-access-ltzcd\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.658752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.658705 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27dab51-2202-47b7-8081-28b6370de6e0-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.658752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.658715 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27dab51-2202-47b7-8081-28b6370de6e0-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.658752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.658722 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27dab51-2202-47b7-8081-28b6370de6e0-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.739949 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:07.739921 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202f1f31-21e9-4309-be30-7807ac06e376" path="/var/lib/kubelet/pods/202f1f31-21e9-4309-be30-7807ac06e376/volumes" Oct 30 02:02:08.634142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:08.630681 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" Oct 30 02:02:08.634142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:08.631743 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4" event={"ID":"a27dab51-2202-47b7-8081-28b6370de6e0","Type":"ContainerDied","Data":"53073857dbabf3d799b7961e761798652776d8d069cab4ba60e8a7fd29ef136d"} Oct 30 02:02:08.634142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:08.631851 6275 scope.go:122] "RemoveContainer" containerID="ff95ece34520d73b3a97f26ba32b14f3a52751feb8c265f1bb66211133aaeda2" Oct 30 02:02:08.712289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:08.712249 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4"] Oct 30 02:02:08.720510 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:08.720480 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-ppqx4"] Oct 30 02:02:09.741634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.741604 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" path="/var/lib/kubelet/pods/a27dab51-2202-47b7-8081-28b6370de6e0/volumes" Oct 30 02:02:09.943496 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.943460 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2"] Oct 30 02:02:09.944233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944212 6275 cpu_manager.go:414] "containerMap: removing container" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" Oct 30 02:02:09.944233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944232 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" Oct 30 02:02:09.944411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944269 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" Oct 30 02:02:09.944411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944277 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" Oct 30 02:02:09.944411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944298 6275 cpu_manager.go:414] "containerMap: removing container" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" Oct 30 02:02:09.944411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944306 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" Oct 30 02:02:09.944631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944505 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" Oct 30 02:02:09.944631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.944522 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27dab51-2202-47b7-8081-28b6370de6e0" containerName="istio-proxy" Oct 30 02:02:09.950012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.949972 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:09.968036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.968010 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2"] Oct 30 02:02:09.973917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:09.973892 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-70a069-dockercfg-mlnfb\"" Oct 30 02:02:10.064146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.063137 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-70a069\"" Oct 30 02:02:10.068144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.064955 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.068144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.065003 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxf9\" (UniqueName: \"kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9\") pod \"payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.167294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.167209 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.167294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.167266 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxf9\" (UniqueName: \"kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9\") pod \"payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.181337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.168808 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.219144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.212213 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxf9\" (UniqueName: \"kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9\") pod \"payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.268678 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.268640 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:10.645241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:10.644482 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2"] Oct 30 02:02:11.710206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:11.710169 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" event={"ID":"cf8f9e34-a798-476c-952f-a85ba336d1a2","Type":"ContainerStarted","Data":"47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3"} Oct 30 02:02:11.710614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:11.710350 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:11.710614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:11.710397 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" event={"ID":"cf8f9e34-a798-476c-952f-a85ba336d1a2","Type":"ContainerStarted","Data":"fe2d7200e54811fecb14cf5b4f21cfefbdb3e8751db0978568c48c077a6a145c"} Oct 30 02:02:11.750843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:11.750794 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" podStartSLOduration=2.750783361 podStartE2EDuration="2.750783361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:11.749419669 +0000 UTC m=+1960.141227303" watchObservedRunningTime="2026-10-30 02:02:11.750783361 +0000 UTC m=+1960.142590994" Oct 30 02:02:14.937032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.932628 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7"] Oct 30 02:02:14.937032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.933839 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="202f1f31-21e9-4309-be30-7807ac06e376" containerName="istio-proxy" Oct 30 02:02:14.954523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.938895 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7"] Oct 30 02:02:14.954523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.939009 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:14.954523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.953803 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:14.954523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.953853 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfx5d\" (UniqueName: \"kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d\") pod \"payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:14.996094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:14.992446 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ea4bc0\"" Oct 30 02:02:15.005422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.005392 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ea4bc0-dockercfg-jf7l2\"" Oct 30 02:02:15.028557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.028527 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:15.047909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.047879 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:15.062201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.062164 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:15.065175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.065146 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfx5d\" (UniqueName: \"kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d\") pod \"payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:15.065852 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.065036 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:15.087594 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.087562 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfx5d\" (UniqueName: \"kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d\") pod \"payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:15.310395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.310362 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:15.621773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.621748 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7"] Oct 30 02:02:15.624450 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:02:15.624420 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8663dd2_2782_4466_9e62_621cd7380d64.slice/crio-23183a618e965d06646ec528b00473d3804682f716929dafd2011e6765c01669 WatchSource:0}: Error finding container 23183a618e965d06646ec528b00473d3804682f716929dafd2011e6765c01669: Status 404 returned error can't find the container with id 23183a618e965d06646ec528b00473d3804682f716929dafd2011e6765c01669 Oct 30 02:02:15.743287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.743252 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" event={"ID":"d8663dd2-2782-4466-9e62-621cd7380d64","Type":"ContainerStarted","Data":"0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6"} Oct 30 02:02:15.743482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.743307 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:15.743482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.743326 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" event={"ID":"d8663dd2-2782-4466-9e62-621cd7380d64","Type":"ContainerStarted","Data":"23183a618e965d06646ec528b00473d3804682f716929dafd2011e6765c01669"} Oct 30 02:02:15.760695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.760666 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:02:15.789826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:15.789772 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" podStartSLOduration=1.789757248 podStartE2EDuration="1.789757248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:15.783935625 +0000 UTC m=+1964.175743258" watchObservedRunningTime="2026-10-30 02:02:15.789757248 +0000 UTC m=+1964.181564889" Oct 30 02:02:16.694366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:16.694333 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn"] Oct 30 02:02:16.694772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:16.694531 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" podUID="efe92a87-64a0-4d6f-9ef6-06680630a68c" containerName="payload-processing" containerID="cri-o://7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7" gracePeriod=30 Oct 30 02:02:16.701472 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:16.701222 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:02:16.712549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:16.712517 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt"] Oct 30 02:02:16.712759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:16.712731 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" podUID="2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" containerName="payload-pre-processing" containerID="cri-o://11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746" gracePeriod=30 Oct 30 02:02:17.016259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.016232 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:02:17.153826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.147222 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume\") pod \"efe92a87-64a0-4d6f-9ef6-06680630a68c\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " Oct 30 02:02:17.153826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.147295 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52\" (UniqueName: \"kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52\") pod \"efe92a87-64a0-4d6f-9ef6-06680630a68c\" (UID: \"efe92a87-64a0-4d6f-9ef6-06680630a68c\") " Oct 30 02:02:17.153826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.152005 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume" pod "efe92a87-64a0-4d6f-9ef6-06680630a68c" (UID: "efe92a87-64a0-4d6f-9ef6-06680630a68c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:17.229985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.229949 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52" pod "efe92a87-64a0-4d6f-9ef6-06680630a68c" (UID: "efe92a87-64a0-4d6f-9ef6-06680630a68c"). InnerVolumeSpecName "kube-api-access-pmn52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:17.237330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.235892 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:02:17.249251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.249218 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx\") pod \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " Oct 30 02:02:17.249688 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.249665 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume\") pod \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\" (UID: \"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e\") " Oct 30 02:02:17.250488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.250463 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe92a87-64a0-4d6f-9ef6-06680630a68c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.250615 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.250602 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn52\" (UniqueName: \"kubernetes.io/projected/efe92a87-64a0-4d6f-9ef6-06680630a68c-kube-api-access-pmn52\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.251508 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.251484 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume" pod "2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" (UID: "2c0bac3f-02d9-44a0-a8cf-b91d9369e61e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:17.258318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.258267 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx" pod "2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" (UID: "2c0bac3f-02d9-44a0-a8cf-b91d9369e61e"). InnerVolumeSpecName "kube-api-access-22tjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:17.352616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.352575 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.352616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.352607 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e-kube-api-access-22tjx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.761779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.761754 6275 generic.go:356] "Generic (PLEG): container finished" podID="2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" containerID="11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746" exitCode=0 Oct 30 02:02:17.762194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.761790 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" Oct 30 02:02:17.762194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.761836 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" event={"ID":"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e","Type":"ContainerDied","Data":"11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746"} Oct 30 02:02:17.762194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.761894 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt" event={"ID":"2c0bac3f-02d9-44a0-a8cf-b91d9369e61e","Type":"ContainerDied","Data":"56a39bd9ae4cdbd848aa745b88b032d2cdb1643a2eab8f061b65a3ddc493d8f6"} Oct 30 02:02:17.762194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.761926 6275 scope.go:122] "RemoveContainer" containerID="11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746" Oct 30 02:02:17.763311 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.763291 6275 generic.go:356] "Generic (PLEG): container finished" podID="efe92a87-64a0-4d6f-9ef6-06680630a68c" containerID="7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7" exitCode=0 Oct 30 02:02:17.763400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.763338 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" Oct 30 02:02:17.763400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.763357 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" event={"ID":"efe92a87-64a0-4d6f-9ef6-06680630a68c","Type":"ContainerDied","Data":"7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7"} Oct 30 02:02:17.763400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.763388 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn" event={"ID":"efe92a87-64a0-4d6f-9ef6-06680630a68c","Type":"ContainerDied","Data":"42ca35e445e09859ad9b4224729bd84f755c9c1c344872b609b0b86ca0e67994"} Oct 30 02:02:17.772492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.772477 6275 scope.go:122] "RemoveContainer" containerID="11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746" Oct 30 02:02:17.772769 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:17.772750 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746\": container with ID starting with 11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746 not found: ID does not exist" containerID="11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746" Oct 30 02:02:17.772832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.772786 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746"} err="failed to get container status \"11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746\": rpc error: code = NotFound desc = could not find container \"11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746\": container with ID starting with 11663c0b88858f0c6a1ca2653047b38551bafd75a1bfb8c4081db54dd9aeb746 not found: ID does not exist" Oct 30 02:02:17.772832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.772804 6275 scope.go:122] "RemoveContainer" containerID="7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7" Oct 30 02:02:17.782372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.782343 6275 scope.go:122] "RemoveContainer" containerID="7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7" Oct 30 02:02:17.782622 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:17.782603 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7\": container with ID starting with 7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7 not found: ID does not exist" containerID="7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7" Oct 30 02:02:17.782678 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.782628 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7"} err="failed to get container status \"7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7\": rpc error: code = NotFound desc = could not find container \"7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7\": container with ID starting with 7c80a58bda13134a79d595c39075942b8b686a508200278df9b3bc41909abaf7 not found: ID does not exist" Oct 30 02:02:17.810002 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.809936 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt"] Oct 30 02:02:17.822330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.821197 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-8bfd75f5-kpldt"] Oct 30 02:02:17.844815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.844790 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn"] Oct 30 02:02:17.848578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:17.848553 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-c8cc8fb7b-kxfjn"] Oct 30 02:02:19.743911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:19.743873 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" path="/var/lib/kubelet/pods/2c0bac3f-02d9-44a0-a8cf-b91d9369e61e/volumes" Oct 30 02:02:19.744530 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:19.744507 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe92a87-64a0-4d6f-9ef6-06680630a68c" path="/var/lib/kubelet/pods/efe92a87-64a0-4d6f-9ef6-06680630a68c/volumes" Oct 30 02:02:21.684309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:21.684278 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js"] Oct 30 02:02:21.684765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:21.684526 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" containerID="cri-o://142bcfafbf570ba85490d6be3ea9aa12fbe9a026cf6705a20d97e838d67db0a8" gracePeriod=30 Oct 30 02:02:22.720922 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:22.720893 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:02:24.708532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.708499 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v"] Oct 30 02:02:24.709244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.709228 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" containerName="payload-pre-processing" Oct 30 02:02:24.709299 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.709246 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" containerName="payload-pre-processing" Oct 30 02:02:24.709299 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.709269 6275 cpu_manager.go:414] "containerMap: removing container" podUID="efe92a87-64a0-4d6f-9ef6-06680630a68c" containerName="payload-processing" Oct 30 02:02:24.709299 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.709274 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe92a87-64a0-4d6f-9ef6-06680630a68c" containerName="payload-processing" Oct 30 02:02:24.709396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.709390 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0bac3f-02d9-44a0-a8cf-b91d9369e61e" containerName="payload-pre-processing" Oct 30 02:02:24.709429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.709400 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe92a87-64a0-4d6f-9ef6-06680630a68c" containerName="payload-processing" Oct 30 02:02:24.712904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.712884 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.726052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.726030 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-168989-serving-cert\"" Oct 30 02:02:24.735361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.735334 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v"] Oct 30 02:02:24.793543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.793507 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmgp\" (UniqueName: \"kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.793737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.793633 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.793737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.793684 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.793900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.793875 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.899703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.899656 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.899920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.899808 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmgp\" (UniqueName: \"kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.899987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.899919 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.899987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.899978 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.901785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.901728 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.913904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.906546 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.916660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.914645 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:24.924698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:24.924675 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmgp\" (UniqueName: \"kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp\") pod \"maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:25.027227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.027201 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:25.335006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.334818 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v"] Oct 30 02:02:25.338070 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:02:25.338042 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a043753_45b2_4147_b917_ef00e11ce6d8.slice/crio-67728531b3f7d02402c0ea760f79399304a87fa2b474d3430f5546e9140367c6 WatchSource:0}: Error finding container 67728531b3f7d02402c0ea760f79399304a87fa2b474d3430f5546e9140367c6: Status 404 returned error can't find the container with id 67728531b3f7d02402c0ea760f79399304a87fa2b474d3430f5546e9140367c6 Oct 30 02:02:25.864381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.863406 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:02:25.864381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.863487 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:02:25.898849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.898808 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" event={"ID":"3a043753-45b2-4147-b917-ef00e11ce6d8","Type":"ContainerStarted","Data":"c6e858d98dc8a7139a53ba018a858a05829c057114de6aacf2b3b5adeeec3922"} Oct 30 02:02:25.899047 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.898863 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" event={"ID":"3a043753-45b2-4147-b917-ef00e11ce6d8","Type":"ContainerStarted","Data":"67728531b3f7d02402c0ea760f79399304a87fa2b474d3430f5546e9140367c6"} Oct 30 02:02:25.899797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.899770 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:25.942181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:25.942061 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" podStartSLOduration=1.61977043 podStartE2EDuration="1.94204767s" totalImagesPullingTime="322.27724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:25.938955955 +0000 UTC m=+1974.330763588" watchObservedRunningTime="2026-10-30 02:02:25.94204767 +0000 UTC m=+1974.333855302" Oct 30 02:02:26.758798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:26.758766 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:02:26.913347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:26.913103 6275 generic.go:356] "Generic (PLEG): container finished" podID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerID="142bcfafbf570ba85490d6be3ea9aa12fbe9a026cf6705a20d97e838d67db0a8" exitCode=0 Oct 30 02:02:26.913347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:26.913176 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" event={"ID":"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4","Type":"ContainerDied","Data":"142bcfafbf570ba85490d6be3ea9aa12fbe9a026cf6705a20d97e838d67db0a8"} Oct 30 02:02:26.913347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:26.913237 6275 scope.go:122] "RemoveContainer" containerID="70f492442de2b0d163806f289b88bf292ee9024a4edde51847a67c23bfce2c35" Oct 30 02:02:26.980351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:26.980322 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 02:02:27.044536 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.044496 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.044727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.044610 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.044799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.044772 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.044860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.044827 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.044928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.044857 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.045102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.044818 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:27.045194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045054 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:27.045277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045258 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.045338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045307 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.045404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045248 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:27.045404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045356 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.045512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045440 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs\") pod \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\" (UID: \"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4\") " Oct 30 02:02:27.045512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.045491 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:27.046564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.046460 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.046564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.046488 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.046564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.046502 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.046564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.046514 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.046564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.046523 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:27.047896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.047863 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:27.048158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.048101 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:27.048232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.048168 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "kube-api-access-jqnk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:27.048677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.048653 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token" pod "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" (UID: "dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:27.147867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.147835 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.147867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.147863 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqnk5\" (UniqueName: \"kubernetes.io/projected/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-kube-api-access-jqnk5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.147867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.147874 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.148132 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.147883 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.148132 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.147891 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:27.921688 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.921659 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" Oct 30 02:02:27.922199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.921700 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js" event={"ID":"dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4","Type":"ContainerDied","Data":"8c693b337edef2bebe4dbbe607236ed33b28c687989529217bae97ae0a3848a8"} Oct 30 02:02:27.922199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.921758 6275 scope.go:122] "RemoveContainer" containerID="142bcfafbf570ba85490d6be3ea9aa12fbe9a026cf6705a20d97e838d67db0a8" Oct 30 02:02:27.981400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.981367 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js"] Oct 30 02:02:27.991463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:27.991441 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-954773-openshift-default-f9885dc7-ln5js"] Oct 30 02:02:29.739945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.739914 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" path="/var/lib/kubelet/pods/dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4/volumes" Oct 30 02:02:29.756904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.756876 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w"] Oct 30 02:02:29.757629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.757611 6275 cpu_manager.go:414] "containerMap: removing container" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" Oct 30 02:02:29.757716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.757631 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" Oct 30 02:02:29.757716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.757656 6275 cpu_manager.go:414] "containerMap: removing container" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" Oct 30 02:02:29.757716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.757662 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" Oct 30 02:02:29.757829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.757774 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" Oct 30 02:02:29.757829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.757784 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0c2dd5-12e1-4fbd-ab38-06abc45df4b4" containerName="istio-proxy" Oct 30 02:02:29.762985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.762965 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:29.776526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.776508 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-70a069-serving-cert\"" Oct 30 02:02:29.793078 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.793049 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w"] Oct 30 02:02:29.888463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.888430 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:29.888660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.888640 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jm6t\" (UniqueName: \"kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:29.888811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.888796 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:29.888880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.888842 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.991559 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.991760 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jm6t\" (UniqueName: \"kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.991912 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:29.991981 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:30.001998 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:30.002278 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-serving-cert: secret "maas-api-e2e-praxis-a-70a069-serving-cert" not found Oct 30 02:02:30.024201 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:30.002376 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls podName:33440bce-816d-42d3-8d44-7fb5c1cd180e nodeName:}" failed. No retries permitted until 2026-10-30 02:02:30.502356667 +0000 UTC m=+1978.894164281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls") pod "maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" (UID: "33440bce-816d-42d3-8d44-7fb5c1cd180e") : secret "maas-api-e2e-praxis-a-70a069-serving-cert" not found Oct 30 02:02:30.049516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:30.044553 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.049516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:30.047967 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jm6t\" (UniqueName: \"kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.509154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:30.509109 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.511581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:30.511555 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:30.676503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:30.676456 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:31.081751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:31.081727 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w"] Oct 30 02:02:31.084299 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:02:31.084272 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33440bce_816d_42d3_8d44_7fb5c1cd180e.slice/crio-23fa4eea0f06cc5d5a63275371c105698f5aa5eac48ac8a7958f4463d411a4ee WatchSource:0}: Error finding container 23fa4eea0f06cc5d5a63275371c105698f5aa5eac48ac8a7958f4463d411a4ee: Status 404 returned error can't find the container with id 23fa4eea0f06cc5d5a63275371c105698f5aa5eac48ac8a7958f4463d411a4ee Oct 30 02:02:32.050012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:32.049976 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" event={"ID":"33440bce-816d-42d3-8d44-7fb5c1cd180e","Type":"ContainerStarted","Data":"2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2"} Oct 30 02:02:32.050231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:32.050036 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" event={"ID":"33440bce-816d-42d3-8d44-7fb5c1cd180e","Type":"ContainerStarted","Data":"23fa4eea0f06cc5d5a63275371c105698f5aa5eac48ac8a7958f4463d411a4ee"} Oct 30 02:02:32.050231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:32.050065 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:32.093229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:32.093176 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" podStartSLOduration=2.763313952 podStartE2EDuration="3.09316269s" totalImagesPullingTime="329.848738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:32.08961856 +0000 UTC m=+1980.481426193" watchObservedRunningTime="2026-10-30 02:02:32.09316269 +0000 UTC m=+1980.484970326" Oct 30 02:02:32.928722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:32.928693 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:02:33.528411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:33.528372 6275 scope.go:122] "RemoveContainer" containerID="51acc65621b498f63db71712dfe2fd09836604b6c4b487a399478a4d39e51175" Oct 30 02:02:37.025187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.025154 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf"] Oct 30 02:02:37.025674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.025411 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" podUID="586b35e3-7fb1-4846-a055-126f5c58f30f" containerName="payload-processing" containerID="cri-o://c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c" gracePeriod=30 Oct 30 02:02:37.049163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.047710 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp"] Oct 30 02:02:37.049163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.047956 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" podUID="76274492-bf77-417d-9b88-59ca6f344e48" containerName="payload-pre-processing" containerID="cri-o://e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b" gracePeriod=30 Oct 30 02:02:37.567134 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.567089 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:02:37.608279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.608246 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:02:37.672694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.672649 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume\") pod \"76274492-bf77-417d-9b88-59ca6f344e48\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " Oct 30 02:02:37.672870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.672802 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume\") pod \"586b35e3-7fb1-4846-a055-126f5c58f30f\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " Oct 30 02:02:37.673014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.672995 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm\" (UniqueName: \"kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm\") pod \"76274492-bf77-417d-9b88-59ca6f344e48\" (UID: \"76274492-bf77-417d-9b88-59ca6f344e48\") " Oct 30 02:02:37.673091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.673043 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8\" (UniqueName: \"kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8\") pod \"586b35e3-7fb1-4846-a055-126f5c58f30f\" (UID: \"586b35e3-7fb1-4846-a055-126f5c58f30f\") " Oct 30 02:02:37.673196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.673160 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume" pod "76274492-bf77-417d-9b88-59ca6f344e48" (UID: "76274492-bf77-417d-9b88-59ca6f344e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:37.673260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.673223 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume" pod "586b35e3-7fb1-4846-a055-126f5c58f30f" (UID: "586b35e3-7fb1-4846-a055-126f5c58f30f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:37.673674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.673645 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76274492-bf77-417d-9b88-59ca6f344e48-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.673785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.673678 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586b35e3-7fb1-4846-a055-126f5c58f30f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.675070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.675045 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm" pod "76274492-bf77-417d-9b88-59ca6f344e48" (UID: "76274492-bf77-417d-9b88-59ca6f344e48"). InnerVolumeSpecName "kube-api-access-6p6cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:37.675900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.675879 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8" pod "586b35e3-7fb1-4846-a055-126f5c58f30f" (UID: "586b35e3-7fb1-4846-a055-126f5c58f30f"). InnerVolumeSpecName "kube-api-access-62sh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:37.810194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.791275 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p6cm\" (UniqueName: \"kubernetes.io/projected/76274492-bf77-417d-9b88-59ca6f344e48-kube-api-access-6p6cm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.810194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:37.791309 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62sh8\" (UniqueName: \"kubernetes.io/projected/586b35e3-7fb1-4846-a055-126f5c58f30f-kube-api-access-62sh8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:02:38.069388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.069355 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:02:38.115493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.115461 6275 generic.go:356] "Generic (PLEG): container finished" podID="586b35e3-7fb1-4846-a055-126f5c58f30f" containerID="c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c" exitCode=0 Oct 30 02:02:38.115631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.115514 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" Oct 30 02:02:38.115631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.115531 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" event={"ID":"586b35e3-7fb1-4846-a055-126f5c58f30f","Type":"ContainerDied","Data":"c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c"} Oct 30 02:02:38.115631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.115582 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf" event={"ID":"586b35e3-7fb1-4846-a055-126f5c58f30f","Type":"ContainerDied","Data":"4991781aed494379cb19e79c66cebff4035b412f8b5d5abcf01f28ac075b2370"} Oct 30 02:02:38.115631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.115614 6275 scope.go:122] "RemoveContainer" containerID="c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c" Oct 30 02:02:38.116996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.116972 6275 generic.go:356] "Generic (PLEG): container finished" podID="76274492-bf77-417d-9b88-59ca6f344e48" containerID="e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b" exitCode=0 Oct 30 02:02:38.117073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.117027 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" Oct 30 02:02:38.117073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.117060 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" event={"ID":"76274492-bf77-417d-9b88-59ca6f344e48","Type":"ContainerDied","Data":"e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b"} Oct 30 02:02:38.117491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.117112 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp" event={"ID":"76274492-bf77-417d-9b88-59ca6f344e48","Type":"ContainerDied","Data":"76871a6fc62303ab1450baf95709e5aca9f8257b4f2d10ecd365f8b2db60752a"} Oct 30 02:02:38.127369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.127343 6275 scope.go:122] "RemoveContainer" containerID="c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c" Oct 30 02:02:38.127764 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:38.127726 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c\": container with ID starting with c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c not found: ID does not exist" containerID="c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c" Oct 30 02:02:38.127895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.127773 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c"} err="failed to get container status \"c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c\": rpc error: code = NotFound desc = could not find container \"c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c\": container with ID starting with c5c10cd9b5a393e253845f36844260076a21f5e37326782f3aa52e0f7ed41f0c not found: ID does not exist" Oct 30 02:02:38.127895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.127798 6275 scope.go:122] "RemoveContainer" containerID="e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b" Oct 30 02:02:38.140327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.140298 6275 scope.go:122] "RemoveContainer" containerID="e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b" Oct 30 02:02:38.140629 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:38.140603 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b\": container with ID starting with e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b not found: ID does not exist" containerID="e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b" Oct 30 02:02:38.140719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.140637 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b"} err="failed to get container status \"e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b\": rpc error: code = NotFound desc = could not find container \"e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b\": container with ID starting with e171f34b4bae5b2f6e3a0b1b39ea2033b6d4a543384c8a0c5645ee31207fd66b not found: ID does not exist" Oct 30 02:02:38.172762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.172732 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp"] Oct 30 02:02:38.177745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.177719 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-7cccb6849f-dv8mp"] Oct 30 02:02:38.207691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.207663 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf"] Oct 30 02:02:38.212186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:38.212163 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38bc0d-7cd6f977b4-ng4qf"] Oct 30 02:02:39.738814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:39.738773 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586b35e3-7fb1-4846-a055-126f5c58f30f" path="/var/lib/kubelet/pods/586b35e3-7fb1-4846-a055-126f5c58f30f/volumes" Oct 30 02:02:39.739391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:39.739371 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76274492-bf77-417d-9b88-59ca6f344e48" path="/var/lib/kubelet/pods/76274492-bf77-417d-9b88-59ca6f344e48/volumes" Oct 30 02:02:46.698467 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:02:46.698385 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:02:55.097910 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.097873 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v"] Oct 30 02:02:55.098936 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.098912 6275 cpu_manager.go:414] "containerMap: removing container" podUID="586b35e3-7fb1-4846-a055-126f5c58f30f" containerName="payload-processing" Oct 30 02:02:55.098936 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.098937 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586b35e3-7fb1-4846-a055-126f5c58f30f" containerName="payload-processing" Oct 30 02:02:55.099107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.098979 6275 cpu_manager.go:414] "containerMap: removing container" podUID="76274492-bf77-417d-9b88-59ca6f344e48" containerName="payload-pre-processing" Oct 30 02:02:55.099107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.098989 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76274492-bf77-417d-9b88-59ca6f344e48" containerName="payload-pre-processing" Oct 30 02:02:55.099227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.099177 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="586b35e3-7fb1-4846-a055-126f5c58f30f" containerName="payload-processing" Oct 30 02:02:55.099227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.099199 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="76274492-bf77-417d-9b88-59ca6f344e48" containerName="payload-pre-processing" Oct 30 02:02:55.103412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.103388 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.120660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.120634 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v"] Oct 30 02:02:55.184145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.167463 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2979be-dockercfg-8cx86\"" Oct 30 02:02:55.205197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.184578 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2979be\"" Oct 30 02:02:55.234148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.208054 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmfk\" (UniqueName: \"kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk\") pod \"payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.234148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.218294 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.331309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.328019 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmfk\" (UniqueName: \"kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk\") pod \"payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.331807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.331773 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.333334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.333308 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.362049 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.361987 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmfk\" (UniqueName: \"kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk\") pod \"payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.420036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.420004 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:55.726347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:55.726267 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v"] Oct 30 02:02:56.376134 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:56.376081 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" event={"ID":"2fc4203c-bc41-4b06-9175-f43ccf6e7a94","Type":"ContainerStarted","Data":"4218302e0ec8b00278f66d67e5f85096a84a4b2f789912acd092d2975d3183f0"} Oct 30 02:02:56.376590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:56.376152 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" event={"ID":"2fc4203c-bc41-4b06-9175-f43ccf6e7a94","Type":"ContainerStarted","Data":"d2f580de500467b096211e546c21f79d8a440c2b133c38cf08a20c6c93dffdd9"} Oct 30 02:02:56.376590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:56.376244 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:02:56.414737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:02:56.414687 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" podStartSLOduration=1.414673263 podStartE2EDuration="1.414673263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:56.413772598 +0000 UTC m=+2004.805580233" watchObservedRunningTime="2026-10-30 02:02:56.414673263 +0000 UTC m=+2004.806480896" Oct 30 02:03:03.991678 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:03.991577 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb"] Oct 30 02:03:03.999756 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:03.999724 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.055991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.055958 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb"] Oct 30 02:03:04.069229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.069195 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-70a069-openshift-default-dockercfg-2bzd9\"" Oct 30 02:03:04.101713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.101679 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.101876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.101751 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.101989 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.102064 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.102150 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.102178 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.102201 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.102256 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jpp\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.102289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.102281 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.203594 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203557 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.203765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203624 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.203765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203661 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.203765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203681 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.203765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203739 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.203765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203756 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203771 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203810 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jpp\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.203861 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.204386 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.204390 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.204451 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.204819 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.204995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.204848 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.206267 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.206244 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.206587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.206565 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.225110 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.225086 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.225513 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.225493 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jpp\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.378227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.378191 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:04.710541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.710505 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb"] Oct 30 02:03:04.712007 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:03:04.711979 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6319398_2ee5_4eed_baae_2e6fe7530c9d.slice/crio-76413ed55d748a6be59903dc9bdc1351a856c2c3f42b553952ffd5a6fd70dc62 WatchSource:0}: Error finding container 76413ed55d748a6be59903dc9bdc1351a856c2c3f42b553952ffd5a6fd70dc62: Status 404 returned error can't find the container with id 76413ed55d748a6be59903dc9bdc1351a856c2c3f42b553952ffd5a6fd70dc62 Oct 30 02:03:04.715043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.715006 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:04.715179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.715090 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:04.715179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:04.715155 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:05.500136 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:05.500089 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" event={"ID":"a6319398-2ee5-4eed-baae-2e6fe7530c9d","Type":"ContainerStarted","Data":"7cef8c2a3b68e6b8c02365106c694a8e76006c8cc87fafe7cad91ff982a7ae9f"} Oct 30 02:03:05.500523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:05.500180 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" event={"ID":"a6319398-2ee5-4eed-baae-2e6fe7530c9d","Type":"ContainerStarted","Data":"76413ed55d748a6be59903dc9bdc1351a856c2c3f42b553952ffd5a6fd70dc62"} Oct 30 02:03:05.579039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:05.578979 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podStartSLOduration=2.578965074 podStartE2EDuration="2.578965074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:05.566637245 +0000 UTC m=+2013.958444904" watchObservedRunningTime="2026-10-30 02:03:05.578965074 +0000 UTC m=+2013.970772707" Oct 30 02:03:06.378800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:06.378730 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:06.379930 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:06.379904 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:03:06.380000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:06.379958 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 02:03:07.379221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:07.379185 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:03:07.379665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:07.379242 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 02:03:07.405446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:07.405409 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:03:08.378541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.378503 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:03:08.378795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.378552 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 02:03:08.526755 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.526720 6275 generic.go:356] "Generic (PLEG): container finished" podID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerID="398e81107bc9832c90de9435b6b8d0bf35c0859f9b58cfe1d2159a17d689edb2" exitCode=0 Oct 30 02:03:08.527187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.526798 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" event={"ID":"cf4979a4-b725-4497-bc66-6c6779fbcff1","Type":"ContainerDied","Data":"398e81107bc9832c90de9435b6b8d0bf35c0859f9b58cfe1d2159a17d689edb2"} Oct 30 02:03:08.527294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.527279 6275 scope.go:122] "RemoveContainer" containerID="398e81107bc9832c90de9435b6b8d0bf35c0859f9b58cfe1d2159a17d689edb2" Oct 30 02:03:08.529195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.529147 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:08.529322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.529234 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:08.529322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:08.529268 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:09.385147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:09.383579 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:03:09.385147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:09.383681 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 02:03:09.535610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:09.535574 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" event={"ID":"cf4979a4-b725-4497-bc66-6c6779fbcff1","Type":"ContainerStarted","Data":"04a1e312d91e3b59d2714c1998c0b3ae7a5a8a3fa3f2948fc7c76795b5011932"} Oct 30 02:03:10.263145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:10.262599 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:03:10.263145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:10.262731 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 02:03:10.263145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:10.262776 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 02:03:10.385148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:10.383537 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:03:10.385148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:10.383599 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 02:03:11.263926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.263527 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 02:03:11.263926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.263578 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 02:03:11.390810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.388828 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:11.390810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.390401 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:11.391404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.391357 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:03:11.935179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.929171 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf"] Oct 30 02:03:11.973225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.972360 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf"] Oct 30 02:03:11.973225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.972506 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:11.986437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.986410 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-168989-dockercfg-cqrn5\"" Oct 30 02:03:11.987022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:11.987006 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-168989\"" Oct 30 02:03:12.089059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.089020 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r44l2\" (UniqueName: \"kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2\") pod \"payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.089282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.089258 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.190320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.190227 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.190501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.190386 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r44l2\" (UniqueName: \"kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2\") pod \"payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.191117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.191090 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.212711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.212684 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r44l2\" (UniqueName: \"kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2\") pod \"payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.264897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.263643 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 02:03:12.264897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.263705 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 02:03:12.290917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.290882 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:12.629168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:12.629143 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf"] Oct 30 02:03:12.631667 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:03:12.631632 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebe6051_c40d_4051_9349_734c4185cbf4.slice/crio-deb9716ab3ade2822cecdc190d2970eada9b8883d5cc598ff7a2903ed9eee3a2 WatchSource:0}: Error finding container deb9716ab3ade2822cecdc190d2970eada9b8883d5cc598ff7a2903ed9eee3a2: Status 404 returned error can't find the container with id deb9716ab3ade2822cecdc190d2970eada9b8883d5cc598ff7a2903ed9eee3a2 Oct 30 02:03:13.259523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:13.259488 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 02:03:13.259788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:13.259758 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 02:03:13.633259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:13.633159 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" event={"ID":"aebe6051-c40d-4051-9349-734c4185cbf4","Type":"ContainerStarted","Data":"8163ce6f11e600217b36877c73ecb0d996b7d70ce94a7d2d29af39e8af8ed164"} Oct 30 02:03:13.633259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:13.633225 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" event={"ID":"aebe6051-c40d-4051-9349-734c4185cbf4","Type":"ContainerStarted","Data":"deb9716ab3ade2822cecdc190d2970eada9b8883d5cc598ff7a2903ed9eee3a2"} Oct 30 02:03:13.633710 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:13.633332 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:13.696688 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:13.696634 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" podStartSLOduration=2.696620089 podStartE2EDuration="2.696620089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:13.693029598 +0000 UTC m=+2022.084837230" watchObservedRunningTime="2026-10-30 02:03:13.696620089 +0000 UTC m=+2022.088427758" Oct 30 02:03:14.259570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:14.259536 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 02:03:14.259889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:14.259858 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 02:03:15.264710 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:15.264681 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:03:15.265108 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:15.265089 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:03:15.265899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:15.265876 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:03:16.697027 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:03:16.696989 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:03:21.857388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:21.857353 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9"] Oct 30 02:03:21.885854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:21.885821 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b"] Oct 30 02:03:24.666975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:24.666934 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:03:26.858465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:26.858396 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" podUID="f584bb13-4fdb-4327-814e-5a963a88abcc" containerName="payload-processing" containerID="cri-o://b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b" gracePeriod=25 Oct 30 02:03:26.886261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:26.886226 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" podUID="e6580c58-9de6-48e8-95fb-cba6698b0cf9" containerName="payload-pre-processing" containerID="cri-o://79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8" gracePeriod=25 Oct 30 02:03:27.171929 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.171904 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:03:27.205921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.205884 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh\" (UniqueName: \"kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh\") pod \"f584bb13-4fdb-4327-814e-5a963a88abcc\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " Oct 30 02:03:27.206082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.206028 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume\") pod \"f584bb13-4fdb-4327-814e-5a963a88abcc\" (UID: \"f584bb13-4fdb-4327-814e-5a963a88abcc\") " Oct 30 02:03:27.206492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.206463 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume" pod "f584bb13-4fdb-4327-814e-5a963a88abcc" (UID: "f584bb13-4fdb-4327-814e-5a963a88abcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:03:27.206599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.206568 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584bb13-4fdb-4327-814e-5a963a88abcc-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:27.208204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.208179 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh" pod "f584bb13-4fdb-4327-814e-5a963a88abcc" (UID: "f584bb13-4fdb-4327-814e-5a963a88abcc"). InnerVolumeSpecName "kube-api-access-wjlwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:03:27.230589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.230560 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:03:27.307549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.307502 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv\" (UniqueName: \"kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv\") pod \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " Oct 30 02:03:27.307759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.307694 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume\") pod \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\" (UID: \"e6580c58-9de6-48e8-95fb-cba6698b0cf9\") " Oct 30 02:03:27.308152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.308090 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume" pod "e6580c58-9de6-48e8-95fb-cba6698b0cf9" (UID: "e6580c58-9de6-48e8-95fb-cba6698b0cf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:03:27.308330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.308309 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlwh\" (UniqueName: \"kubernetes.io/projected/f584bb13-4fdb-4327-814e-5a963a88abcc-kube-api-access-wjlwh\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:27.308403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.308347 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6580c58-9de6-48e8-95fb-cba6698b0cf9-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:27.309895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.309863 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv" pod "e6580c58-9de6-48e8-95fb-cba6698b0cf9" (UID: "e6580c58-9de6-48e8-95fb-cba6698b0cf9"). InnerVolumeSpecName "kube-api-access-js7bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:03:27.409654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.409574 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7bv\" (UniqueName: \"kubernetes.io/projected/e6580c58-9de6-48e8-95fb-cba6698b0cf9-kube-api-access-js7bv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:27.901527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.901494 6275 generic.go:356] "Generic (PLEG): container finished" podID="e6580c58-9de6-48e8-95fb-cba6698b0cf9" containerID="79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8" exitCode=0 Oct 30 02:03:27.902014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.901555 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" Oct 30 02:03:27.902014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.901577 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" event={"ID":"e6580c58-9de6-48e8-95fb-cba6698b0cf9","Type":"ContainerDied","Data":"79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8"} Oct 30 02:03:27.902014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.901633 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b" event={"ID":"e6580c58-9de6-48e8-95fb-cba6698b0cf9","Type":"ContainerDied","Data":"3904941f5f7268c56da59dc74258763dfbd73dbb6dd8aee2bfbbc0ae06cce34d"} Oct 30 02:03:27.902014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.901666 6275 scope.go:122] "RemoveContainer" containerID="79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8" Oct 30 02:03:27.903332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.903308 6275 generic.go:356] "Generic (PLEG): container finished" podID="f584bb13-4fdb-4327-814e-5a963a88abcc" containerID="b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b" exitCode=0 Oct 30 02:03:27.903468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.903356 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" Oct 30 02:03:27.903468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.903403 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" event={"ID":"f584bb13-4fdb-4327-814e-5a963a88abcc","Type":"ContainerDied","Data":"b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b"} Oct 30 02:03:27.903468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.903452 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9" event={"ID":"f584bb13-4fdb-4327-814e-5a963a88abcc","Type":"ContainerDied","Data":"76addfb29a9e0c2bf385b05293aaccce49166b491c651af4b878011f67be348a"} Oct 30 02:03:27.912358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.912340 6275 scope.go:122] "RemoveContainer" containerID="79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8" Oct 30 02:03:27.912624 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:03:27.912605 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8\": container with ID starting with 79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8 not found: ID does not exist" containerID="79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8" Oct 30 02:03:27.912683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.912632 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8"} err="failed to get container status \"79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8\": rpc error: code = NotFound desc = could not find container \"79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8\": container with ID starting with 79d36fcad07a5dfe2170a218a273a4b86c64aff16a1cec46f90311b1e1363cf8 not found: ID does not exist" Oct 30 02:03:27.912683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.912651 6275 scope.go:122] "RemoveContainer" containerID="b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b" Oct 30 02:03:27.921895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.921878 6275 scope.go:122] "RemoveContainer" containerID="b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b" Oct 30 02:03:27.922172 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:03:27.922156 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b\": container with ID starting with b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b not found: ID does not exist" containerID="b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b" Oct 30 02:03:27.922233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.922179 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b"} err="failed to get container status \"b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b\": rpc error: code = NotFound desc = could not find container \"b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b\": container with ID starting with b4b4029de7e0bb477442f6f7be26b6f212a818fbb59fe6d0723851ef8be1596b not found: ID does not exist" Oct 30 02:03:27.952374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.952341 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9"] Oct 30 02:03:27.957603 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.957576 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-95daa2-56c855c684-r75r9"] Oct 30 02:03:27.989469 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.989444 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b"] Oct 30 02:03:27.992517 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:27.992494 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-95daa2-57d55b5647-x2h5b"] Oct 30 02:03:28.244051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244019 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv"] Oct 30 02:03:28.244706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244691 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f584bb13-4fdb-4327-814e-5a963a88abcc" containerName="payload-processing" Oct 30 02:03:28.244761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244707 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f584bb13-4fdb-4327-814e-5a963a88abcc" containerName="payload-processing" Oct 30 02:03:28.244761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244723 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e6580c58-9de6-48e8-95fb-cba6698b0cf9" containerName="payload-pre-processing" Oct 30 02:03:28.244761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244728 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6580c58-9de6-48e8-95fb-cba6698b0cf9" containerName="payload-pre-processing" Oct 30 02:03:28.244855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244834 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f584bb13-4fdb-4327-814e-5a963a88abcc" containerName="payload-processing" Oct 30 02:03:28.244855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.244843 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6580c58-9de6-48e8-95fb-cba6698b0cf9" containerName="payload-pre-processing" Oct 30 02:03:28.249800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.249775 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.269287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.269254 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0f3cd8-openshift-default-dockercfg-xrwrt\"" Oct 30 02:03:28.272643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.272613 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv"] Oct 30 02:03:28.321011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.320978 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321299 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321412 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321595 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321499 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxg4f\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321619 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321652 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321755 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321791 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.321865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.321824 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429303 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429418 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxg4f\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429454 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429481 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429581 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429609 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429642 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429700 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.429928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.429868 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.430463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.430431 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.446141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.433643 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.446141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.433917 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.458140 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.457718 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.458335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.458316 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.483141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.478892 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.501190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.497879 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.527199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.514768 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.562273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.555344 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxg4f\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f\") pod \"e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.586210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.570460 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:28.975222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.975062 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv"] Oct 30 02:03:28.978097 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:03:28.978068 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c3d3c4_718f_4eef_ae21_124902a580be.slice/crio-81bed210221ac52c5fc37e694935013f7e4600ec4e270ca40ae1d4aecce7e419 WatchSource:0}: Error finding container 81bed210221ac52c5fc37e694935013f7e4600ec4e270ca40ae1d4aecce7e419: Status 404 returned error can't find the container with id 81bed210221ac52c5fc37e694935013f7e4600ec4e270ca40ae1d4aecce7e419 Oct 30 02:03:28.980732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.980700 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:28.980821 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.980764 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:28.980821 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:28.980791 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:03:29.740087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:29.740052 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6580c58-9de6-48e8-95fb-cba6698b0cf9" path="/var/lib/kubelet/pods/e6580c58-9de6-48e8-95fb-cba6698b0cf9/volumes" Oct 30 02:03:29.740565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:29.740550 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f584bb13-4fdb-4327-814e-5a963a88abcc" path="/var/lib/kubelet/pods/f584bb13-4fdb-4327-814e-5a963a88abcc/volumes" Oct 30 02:03:29.925954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:29.925917 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" event={"ID":"18c3d3c4-718f-4eef-ae21-124902a580be","Type":"ContainerStarted","Data":"666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0"} Oct 30 02:03:29.926135 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:29.925970 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" event={"ID":"18c3d3c4-718f-4eef-ae21-124902a580be","Type":"ContainerStarted","Data":"81bed210221ac52c5fc37e694935013f7e4600ec4e270ca40ae1d4aecce7e419"} Oct 30 02:03:29.988522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:29.988467 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podStartSLOduration=1.988453461 podStartE2EDuration="1.988453461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:29.985504731 +0000 UTC m=+2038.377312367" watchObservedRunningTime="2026-10-30 02:03:29.988453461 +0000 UTC m=+2038.380261093" Oct 30 02:03:30.571060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:30.571029 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:30.576743 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:30.576691 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:30.932957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:30.932870 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:30.933774 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:30.933755 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:03:34.451452 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.451231 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77"] Oct 30 02:03:34.490467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.490432 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.536209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.529367 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0f3cd8\"" Oct 30 02:03:34.536209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.529463 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0f3cd8-dockercfg-jrr4c\"" Oct 30 02:03:34.572210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.568645 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9gj\" (UniqueName: \"kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj\") pod \"payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.572210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.568712 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.641238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.627197 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77"] Oct 30 02:03:34.680511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.670335 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9gj\" (UniqueName: \"kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj\") pod \"payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.680511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.670392 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.680511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.672238 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.732153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.730345 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9gj\" (UniqueName: \"kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj\") pod \"payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:34.952590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:34.952557 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:35.259713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:35.259679 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77"] Oct 30 02:03:35.261778 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:03:35.261745 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0752ccc_f33c_41d7_af9a_5df5defff8ab.slice/crio-f07a2d28b47283cf8795eabc8d7dc31b4eea4591eb464304dc2cbe7a680e40da WatchSource:0}: Error finding container f07a2d28b47283cf8795eabc8d7dc31b4eea4591eb464304dc2cbe7a680e40da: Status 404 returned error can't find the container with id f07a2d28b47283cf8795eabc8d7dc31b4eea4591eb464304dc2cbe7a680e40da Oct 30 02:03:36.050080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:36.050047 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" event={"ID":"f0752ccc-f33c-41d7-af9a-5df5defff8ab","Type":"ContainerStarted","Data":"2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9"} Oct 30 02:03:36.050506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:36.050102 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" event={"ID":"f0752ccc-f33c-41d7-af9a-5df5defff8ab","Type":"ContainerStarted","Data":"f07a2d28b47283cf8795eabc8d7dc31b4eea4591eb464304dc2cbe7a680e40da"} Oct 30 02:03:36.050506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:36.050247 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:36.093859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:36.093806 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" podStartSLOduration=2.093789384 podStartE2EDuration="2.093789384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:36.091198275 +0000 UTC m=+2044.483005909" watchObservedRunningTime="2026-10-30 02:03:36.093789384 +0000 UTC m=+2044.485597019" Oct 30 02:03:37.435473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.435437 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr"] Oct 30 02:03:37.435944 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.435639 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" podUID="ae01f56c-6259-4e01-a320-e5978662deb3" containerName="maas-api" containerID="cri-o://2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2" gracePeriod=30 Oct 30 02:03:37.868146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.867189 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:03:37.982143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.979660 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk\" (UniqueName: \"kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk\") pod \"ae01f56c-6259-4e01-a320-e5978662deb3\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " Oct 30 02:03:37.982143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.979973 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca\") pod \"ae01f56c-6259-4e01-a320-e5978662deb3\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " Oct 30 02:03:37.982143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.980109 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs\") pod \"ae01f56c-6259-4e01-a320-e5978662deb3\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " Oct 30 02:03:37.982143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.980190 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\") pod \"ae01f56c-6259-4e01-a320-e5978662deb3\" (UID: \"ae01f56c-6259-4e01-a320-e5978662deb3\") " Oct 30 02:03:37.982518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:37.982375 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca" pod "ae01f56c-6259-4e01-a320-e5978662deb3" (UID: "ae01f56c-6259-4e01-a320-e5978662deb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:03:38.001437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.001301 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk" pod "ae01f56c-6259-4e01-a320-e5978662deb3" (UID: "ae01f56c-6259-4e01-a320-e5978662deb3"). InnerVolumeSpecName "kube-api-access-wdblk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:03:38.001437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.001394 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls" pod "ae01f56c-6259-4e01-a320-e5978662deb3" (UID: "ae01f56c-6259-4e01-a320-e5978662deb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:03:38.033231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.027675 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs" pod "ae01f56c-6259-4e01-a320-e5978662deb3" (UID: "ae01f56c-6259-4e01-a320-e5978662deb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:03:38.093192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.078771 6275 generic.go:356] "Generic (PLEG): container finished" podID="ae01f56c-6259-4e01-a320-e5978662deb3" containerID="2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2" exitCode=0 Oct 30 02:03:38.093192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.078945 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" event={"ID":"ae01f56c-6259-4e01-a320-e5978662deb3","Type":"ContainerDied","Data":"2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2"} Oct 30 02:03:38.093192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.078997 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" event={"ID":"ae01f56c-6259-4e01-a320-e5978662deb3","Type":"ContainerDied","Data":"32861d74b79b6c6e2b868828d47e37cca6cef64aa5b59eebee7253c6f0b42bb1"} Oct 30 02:03:38.093192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.079026 6275 scope.go:122] "RemoveContainer" containerID="2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2" Oct 30 02:03:38.093192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.079222 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr" Oct 30 02:03:38.096714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.095957 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae01f56c-6259-4e01-a320-e5978662deb3-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:38.096714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.095993 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:38.096714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.096008 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae01f56c-6259-4e01-a320-e5978662deb3-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:38.096714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.096022 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdblk\" (UniqueName: \"kubernetes.io/projected/ae01f56c-6259-4e01-a320-e5978662deb3-kube-api-access-wdblk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:03:38.109080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.109053 6275 scope.go:122] "RemoveContainer" containerID="2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2" Oct 30 02:03:38.109435 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:03:38.109413 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2\": container with ID starting with 2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2 not found: ID does not exist" containerID="2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2" Oct 30 02:03:38.109504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.109443 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2"} err="failed to get container status \"2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2\": rpc error: code = NotFound desc = could not find container \"2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2\": container with ID starting with 2a0bf83e02a6416d62c374b2dc0f5840d3649e69b335262a6f0e71e3ffe9edd2 not found: ID does not exist" Oct 30 02:03:38.181912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.181880 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr"] Oct 30 02:03:38.195491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.195458 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-95daa2-6c9f94485-7zdfr"] Oct 30 02:03:38.259647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.259610 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts"] Oct 30 02:03:38.260279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.260265 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ae01f56c-6259-4e01-a320-e5978662deb3" containerName="maas-api" Oct 30 02:03:38.260324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.260280 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae01f56c-6259-4e01-a320-e5978662deb3" containerName="maas-api" Oct 30 02:03:38.260402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.260393 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae01f56c-6259-4e01-a320-e5978662deb3" containerName="maas-api" Oct 30 02:03:38.263847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.263817 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.286009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.285932 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-003ec6-serving-cert\"" Oct 30 02:03:38.310289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.310249 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts"] Oct 30 02:03:38.408859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.408822 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.409048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.408894 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.409048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.408965 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ddv\" (UniqueName: \"kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.409210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.409184 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.515147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.514906 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.515147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.515098 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.515733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.515191 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.515733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.515234 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ddv\" (UniqueName: \"kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.529148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.517113 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.532327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.532292 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.535309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.535266 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.553746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.553669 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ddv\" (UniqueName: \"kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv\") pod \"maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.577155 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.577094 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:38.886704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:38.886677 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts"] Oct 30 02:03:38.889228 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:03:38.889195 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9e3e0a_d578_47c3_9500_3e74eef20ff4.slice/crio-f6eb363aedd92cd4c99de6bf67e0ae35080ab31c39127bff4323a229aa2d986b WatchSource:0}: Error finding container f6eb363aedd92cd4c99de6bf67e0ae35080ab31c39127bff4323a229aa2d986b: Status 404 returned error can't find the container with id f6eb363aedd92cd4c99de6bf67e0ae35080ab31c39127bff4323a229aa2d986b Oct 30 02:03:39.087644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:39.087609 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" event={"ID":"4f9e3e0a-d578-47c3-9500-3e74eef20ff4","Type":"ContainerStarted","Data":"f6eb363aedd92cd4c99de6bf67e0ae35080ab31c39127bff4323a229aa2d986b"} Oct 30 02:03:39.741771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:39.741697 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae01f56c-6259-4e01-a320-e5978662deb3" path="/var/lib/kubelet/pods/ae01f56c-6259-4e01-a320-e5978662deb3/volumes" Oct 30 02:03:40.096032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:40.095987 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" event={"ID":"4f9e3e0a-d578-47c3-9500-3e74eef20ff4","Type":"ContainerStarted","Data":"cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6"} Oct 30 02:03:40.096248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:40.096156 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:40.140432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:40.140386 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" podStartSLOduration=1.758220125 podStartE2EDuration="2.14037428s" totalImagesPullingTime="382.154155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:40.137796567 +0000 UTC m=+2048.529604200" watchObservedRunningTime="2026-10-30 02:03:40.14037428 +0000 UTC m=+2048.532181913" Oct 30 02:03:46.108373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:46.108342 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:03:46.748602 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:03:46.748560 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:03:47.060831 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:47.060803 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:03:57.730797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:57.730761 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8_e181d463-7e41-4707-949a-663cf181149c/payload-processing/0.log" Oct 30 02:03:57.950486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:57.950460 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-qfc9q_e75c1729-dbe8-4486-a74d-fa6611610662/payload-processing/0.log" Oct 30 02:03:58.160397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:58.160367 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-qfc9q_e75c1729-dbe8-4486-a74d-fa6611610662/payload-processing/0.log" Oct 30 02:03:58.343750 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:58.343722 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742_9022ffe1-ea98-4fc1-b4ad-344020ff4a3f/payload-processing/0.log" Oct 30 02:03:58.528347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:58.528319 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742_9022ffe1-ea98-4fc1-b4ad-344020ff4a3f/payload-processing/0.log" Oct 30 02:03:59.717465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:03:59.717429 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7"] Oct 30 02:04:04.492176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:04.492116 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5"] Oct 30 02:04:04.717951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:04.717908 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" podUID="d8663dd2-2782-4466-9e62-621cd7380d64" containerName="payload-processing" containerID="cri-o://0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6" gracePeriod=25 Oct 30 02:04:05.001003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.000974 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:04:05.169764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.169675 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume\") pod \"d8663dd2-2782-4466-9e62-621cd7380d64\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " Oct 30 02:04:05.169764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.169752 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d\" (UniqueName: \"kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d\") pod \"d8663dd2-2782-4466-9e62-621cd7380d64\" (UID: \"d8663dd2-2782-4466-9e62-621cd7380d64\") " Oct 30 02:04:05.170188 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.170157 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume" pod "d8663dd2-2782-4466-9e62-621cd7380d64" (UID: "d8663dd2-2782-4466-9e62-621cd7380d64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:05.170387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.170368 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8663dd2-2782-4466-9e62-621cd7380d64-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:05.172715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.172684 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d" pod "d8663dd2-2782-4466-9e62-621cd7380d64" (UID: "d8663dd2-2782-4466-9e62-621cd7380d64"). InnerVolumeSpecName "kube-api-access-gfx5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:05.272237 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.272205 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfx5d\" (UniqueName: \"kubernetes.io/projected/d8663dd2-2782-4466-9e62-621cd7380d64-kube-api-access-gfx5d\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:05.432804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.432712 6275 generic.go:356] "Generic (PLEG): container finished" podID="d8663dd2-2782-4466-9e62-621cd7380d64" containerID="0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6" exitCode=0 Oct 30 02:04:05.432804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.432761 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" Oct 30 02:04:05.433007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.432800 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" event={"ID":"d8663dd2-2782-4466-9e62-621cd7380d64","Type":"ContainerDied","Data":"0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6"} Oct 30 02:04:05.433007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.432853 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7" event={"ID":"d8663dd2-2782-4466-9e62-621cd7380d64","Type":"ContainerDied","Data":"23183a618e965d06646ec528b00473d3804682f716929dafd2011e6765c01669"} Oct 30 02:04:05.433007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.432875 6275 scope.go:122] "RemoveContainer" containerID="0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6" Oct 30 02:04:05.443562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.443545 6275 scope.go:122] "RemoveContainer" containerID="0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6" Oct 30 02:04:05.443835 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:05.443807 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6\": container with ID starting with 0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6 not found: ID does not exist" containerID="0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6" Oct 30 02:04:05.443901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.443847 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6"} err="failed to get container status \"0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6\": rpc error: code = NotFound desc = could not find container \"0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6\": container with ID starting with 0ac0dfc6bc1e0ee229a69ed351ab1192a25ac88d02b63abea28909144064e0c6 not found: ID does not exist" Oct 30 02:04:05.480019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.479986 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7"] Oct 30 02:04:05.486186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.486160 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea4bc0-6b47d8d584-wgcm7"] Oct 30 02:04:05.898398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:05.898356 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8663dd2-2782-4466-9e62-621cd7380d64" path="/var/lib/kubelet/pods/d8663dd2-2782-4466-9e62-621cd7380d64/volumes" Oct 30 02:04:16.698269 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:16.698181 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:04:19.515115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.515049 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" podUID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" containerName="main" containerID="cri-o://5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc" gracePeriod=45 Oct 30 02:04:19.774459 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.774371 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2"] Oct 30 02:04:19.896245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.896214 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:04:19.975750 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.975705 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs\") pod \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " Oct 30 02:04:19.975920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.975802 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm\") pod \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " Oct 30 02:04:19.975920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.975857 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5\" (UniqueName: \"kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5\") pod \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " Oct 30 02:04:19.976052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.976034 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache\") pod \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " Oct 30 02:04:19.976184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.976165 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir\") pod \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " Oct 30 02:04:19.976276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.976261 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home\") pod \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\" (UID: \"967bb5b5-0617-4eb8-8cd4-0543325c11a3\") " Oct 30 02:04:19.977233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.977205 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home" pod "967bb5b5-0617-4eb8-8cd4-0543325c11a3" (UID: "967bb5b5-0617-4eb8-8cd4-0543325c11a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:19.978428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.978403 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache" pod "967bb5b5-0617-4eb8-8cd4-0543325c11a3" (UID: "967bb5b5-0617-4eb8-8cd4-0543325c11a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:19.978781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.978761 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir" pod "967bb5b5-0617-4eb8-8cd4-0543325c11a3" (UID: "967bb5b5-0617-4eb8-8cd4-0543325c11a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:19.980681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.980628 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs" pod "967bb5b5-0617-4eb8-8cd4-0543325c11a3" (UID: "967bb5b5-0617-4eb8-8cd4-0543325c11a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:19.980917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.980897 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5" pod "967bb5b5-0617-4eb8-8cd4-0543325c11a3" (UID: "967bb5b5-0617-4eb8-8cd4-0543325c11a3"). InnerVolumeSpecName "kube-api-access-rf6g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:19.981184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:19.981163 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm" pod "967bb5b5-0617-4eb8-8cd4-0543325c11a3" (UID: "967bb5b5-0617-4eb8-8cd4-0543325c11a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:20.078668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.078579 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:20.078668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.078608 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf6g5\" (UniqueName: \"kubernetes.io/projected/967bb5b5-0617-4eb8-8cd4-0543325c11a3-kube-api-access-rf6g5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:20.078668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.078617 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:20.078668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.078626 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:20.078668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.078636 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/967bb5b5-0617-4eb8-8cd4-0543325c11a3-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:20.078668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.078644 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/967bb5b5-0617-4eb8-8cd4-0543325c11a3-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:20.769285 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.769238 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" podUID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:04:20.828906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.826675 6275 generic.go:356] "Generic (PLEG): container finished" podID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" containerID="5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc" exitCode=0 Oct 30 02:04:20.828906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.826762 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" event={"ID":"967bb5b5-0617-4eb8-8cd4-0543325c11a3","Type":"ContainerDied","Data":"5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc"} Oct 30 02:04:20.828906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.826797 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" event={"ID":"967bb5b5-0617-4eb8-8cd4-0543325c11a3","Type":"ContainerDied","Data":"8d42f645f98f2b2460206650ed517c36d34893627747a8753cc8fde3eefe9a11"} Oct 30 02:04:20.828906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.826819 6275 scope.go:122] "RemoveContainer" containerID="5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc" Oct 30 02:04:20.828906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.827028 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5" Oct 30 02:04:20.884906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.884831 6275 scope.go:122] "RemoveContainer" containerID="5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc" Oct 30 02:04:20.885979 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:20.885474 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc\": container with ID starting with 5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc not found: ID does not exist" containerID="5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc" Oct 30 02:04:20.885979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.885518 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc"} err="failed to get container status \"5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc\": rpc error: code = NotFound desc = could not find container \"5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc\": container with ID starting with 5f1495e702a0878cfbbb39339dca0e3b6a04909ea152b63bb8c1a575f6c2c1cc not found: ID does not exist" Oct 30 02:04:20.926533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.926489 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5"] Oct 30 02:04:20.931677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:20.931638 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-5c90f5/ipp-route-5c90f5-kserve-7568cdcc97-z48n5"] Oct 30 02:04:21.754448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:21.754410 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" path="/var/lib/kubelet/pods/967bb5b5-0617-4eb8-8cd4-0543325c11a3/volumes" Oct 30 02:04:24.778269 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:24.778166 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" podUID="cf8f9e34-a798-476c-952f-a85ba336d1a2" containerName="payload-pre-processing" containerID="cri-o://47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3" gracePeriod=25 Oct 30 02:04:25.094704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.094673 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:04:25.162250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.162213 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9\" (UniqueName: \"kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9\") pod \"cf8f9e34-a798-476c-952f-a85ba336d1a2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " Oct 30 02:04:25.162435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.162281 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume\") pod \"cf8f9e34-a798-476c-952f-a85ba336d1a2\" (UID: \"cf8f9e34-a798-476c-952f-a85ba336d1a2\") " Oct 30 02:04:25.163000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.162967 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume" pod "cf8f9e34-a798-476c-952f-a85ba336d1a2" (UID: "cf8f9e34-a798-476c-952f-a85ba336d1a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:25.165182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.165093 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9" pod "cf8f9e34-a798-476c-952f-a85ba336d1a2" (UID: "cf8f9e34-a798-476c-952f-a85ba336d1a2"). InnerVolumeSpecName "kube-api-access-7zxf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:25.264892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.264852 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxf9\" (UniqueName: \"kubernetes.io/projected/cf8f9e34-a798-476c-952f-a85ba336d1a2-kube-api-access-7zxf9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:25.264892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.264894 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8f9e34-a798-476c-952f-a85ba336d1a2-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:25.885872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.885832 6275 generic.go:356] "Generic (PLEG): container finished" podID="cf8f9e34-a798-476c-952f-a85ba336d1a2" containerID="47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3" exitCode=0 Oct 30 02:04:25.886362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.885879 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" Oct 30 02:04:25.886362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.885880 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" event={"ID":"cf8f9e34-a798-476c-952f-a85ba336d1a2","Type":"ContainerDied","Data":"47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3"} Oct 30 02:04:25.886362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.885944 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2" event={"ID":"cf8f9e34-a798-476c-952f-a85ba336d1a2","Type":"ContainerDied","Data":"fe2d7200e54811fecb14cf5b4f21cfefbdb3e8751db0978568c48c077a6a145c"} Oct 30 02:04:25.886362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.885975 6275 scope.go:122] "RemoveContainer" containerID="47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3" Oct 30 02:04:25.896355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.896338 6275 scope.go:122] "RemoveContainer" containerID="47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3" Oct 30 02:04:25.896640 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:25.896622 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3\": container with ID starting with 47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3 not found: ID does not exist" containerID="47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3" Oct 30 02:04:25.896734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.896648 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3"} err="failed to get container status \"47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3\": rpc error: code = NotFound desc = could not find container \"47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3\": container with ID starting with 47431018b6b6a590c05c130feb6e83d2b41bfd515efbb4aeef650f92626958c3 not found: ID does not exist" Oct 30 02:04:25.935628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.935591 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2"] Oct 30 02:04:25.945793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:25.945768 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70a069-84d89c7cff-cjrm2"] Oct 30 02:04:26.894602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:26.894565 6275 generic.go:356] "Generic (PLEG): container finished" podID="18c3d3c4-718f-4eef-ae21-124902a580be" containerID="666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0" exitCode=0 Oct 30 02:04:26.895052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:26.894628 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" event={"ID":"18c3d3c4-718f-4eef-ae21-124902a580be","Type":"ContainerDied","Data":"666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0"} Oct 30 02:04:26.895098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:26.895065 6275 scope.go:122] "RemoveContainer" containerID="666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0" Oct 30 02:04:26.896662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:26.896630 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:04:26.896753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:26.896724 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:04:26.896797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:26.896762 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:04:27.758144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:27.756840 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8f9e34-a798-476c-952f-a85ba336d1a2" path="/var/lib/kubelet/pods/cf8f9e34-a798-476c-952f-a85ba336d1a2/volumes" Oct 30 02:04:27.903184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:27.903151 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" event={"ID":"18c3d3c4-718f-4eef-ae21-124902a580be","Type":"ContainerStarted","Data":"b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78"} Oct 30 02:04:28.413109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:28.413072 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77"] Oct 30 02:04:28.572454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:28.571342 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:04:28.578392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:28.573466 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 02:04:28.578392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:28.573537 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 02:04:29.571874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:29.571838 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 02:04:29.572512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:29.571889 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 02:04:30.571335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:30.571302 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 02:04:30.571523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:30.571363 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 02:04:30.671145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:30.669453 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6"] Oct 30 02:04:30.671145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:30.669692 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" containerID="cri-o://df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452" gracePeriod=30 Oct 30 02:04:30.933757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:30.933674 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:04:31.571790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:31.571703 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 02:04:31.571790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:31.571767 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 02:04:32.233683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:32.233652 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:04:32.234662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:32.234637 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:04:32.571567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:32.571535 6275 patch_prober.go:29] interesting pod/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 02:04:32.571814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:32.571791 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 02:04:33.413796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.413722 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" podUID="f0752ccc-f33c-41d7-af9a-5df5defff8ab" containerName="payload-processing" containerID="cri-o://2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9" gracePeriod=25 Oct 30 02:04:33.799653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.799626 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:04:33.914680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.914642 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:04:33.916366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.916340 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:04:33.934104 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.934035 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj\" (UniqueName: \"kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj\") pod \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " Oct 30 02:04:33.934104 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.934076 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume\") pod \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\" (UID: \"f0752ccc-f33c-41d7-af9a-5df5defff8ab\") " Oct 30 02:04:33.934528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.934501 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume" pod "f0752ccc-f33c-41d7-af9a-5df5defff8ab" (UID: "f0752ccc-f33c-41d7-af9a-5df5defff8ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:33.934669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.934653 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0752ccc-f33c-41d7-af9a-5df5defff8ab-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:33.936320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:33.936294 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj" pod "f0752ccc-f33c-41d7-af9a-5df5defff8ab" (UID: "f0752ccc-f33c-41d7-af9a-5df5defff8ab"). InnerVolumeSpecName "kube-api-access-9t9gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:34.036246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.036210 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9gj\" (UniqueName: \"kubernetes.io/projected/f0752ccc-f33c-41d7-af9a-5df5defff8ab-kube-api-access-9t9gj\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:34.117588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.117552 6275 generic.go:356] "Generic (PLEG): container finished" podID="f0752ccc-f33c-41d7-af9a-5df5defff8ab" containerID="2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9" exitCode=0 Oct 30 02:04:34.117757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.117604 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" Oct 30 02:04:34.117757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.117644 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" event={"ID":"f0752ccc-f33c-41d7-af9a-5df5defff8ab","Type":"ContainerDied","Data":"2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9"} Oct 30 02:04:34.117757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.117703 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77" event={"ID":"f0752ccc-f33c-41d7-af9a-5df5defff8ab","Type":"ContainerDied","Data":"f07a2d28b47283cf8795eabc8d7dc31b4eea4591eb464304dc2cbe7a680e40da"} Oct 30 02:04:34.117757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.117731 6275 scope.go:122] "RemoveContainer" containerID="2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9" Oct 30 02:04:34.145319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.145293 6275 scope.go:122] "RemoveContainer" containerID="2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9" Oct 30 02:04:34.145773 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:34.145748 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9\": container with ID starting with 2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9 not found: ID does not exist" containerID="2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9" Oct 30 02:04:34.145883 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.145783 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9"} err="failed to get container status \"2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9\": rpc error: code = NotFound desc = could not find container \"2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9\": container with ID starting with 2a2f594a42f467db851cc7f00ca6805470680f7fbc11e519d486ea5cca5d9fc9 not found: ID does not exist" Oct 30 02:04:34.176777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.176746 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77"] Oct 30 02:04:34.183090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.183064 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-69f455dc4b-zks77"] Oct 30 02:04:34.517695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.517661 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7"] Oct 30 02:04:34.518222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:34.517872 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" containerID="cri-o://8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8" gracePeriod=30 Oct 30 02:04:35.747807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:35.747767 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0752ccc-f33c-41d7-af9a-5df5defff8ab" path="/var/lib/kubelet/pods/f0752ccc-f33c-41d7-af9a-5df5defff8ab/volumes" Oct 30 02:04:35.818625 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:35.818590 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:04:35.818783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:35.818677 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:04:35.952279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:35.952253 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 02:04:36.065272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065237 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065298 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065334 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065362 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065447 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065511 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065646 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065729 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:36.065782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065740 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.065889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065793 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:36.065889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065853 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket\") pod \"d862bc10-3875-4427-9eee-1644c5e8e116\" (UID: \"d862bc10-3875-4427-9eee-1644c5e8e116\") " Oct 30 02:04:36.066005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.065972 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:36.066174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066106 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:36.066287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066262 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:36.066753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066726 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d862bc10-3875-4427-9eee-1644c5e8e116-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.066848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066758 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.066848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066772 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.066848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066787 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.066848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.066800 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.067882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.067855 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "kube-api-access-tsg7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:36.067882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.067872 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:36.068068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.068049 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:36.068787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.068761 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy" pod "d862bc10-3875-4427-9eee-1644c5e8e116" (UID: "d862bc10-3875-4427-9eee-1644c5e8e116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:36.132550 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.132519 6275 generic.go:356] "Generic (PLEG): container finished" podID="d862bc10-3875-4427-9eee-1644c5e8e116" containerID="df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452" exitCode=0 Oct 30 02:04:36.132700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.132581 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" Oct 30 02:04:36.132700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.132605 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" event={"ID":"d862bc10-3875-4427-9eee-1644c5e8e116","Type":"ContainerDied","Data":"df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452"} Oct 30 02:04:36.132700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.132661 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6" event={"ID":"d862bc10-3875-4427-9eee-1644c5e8e116","Type":"ContainerDied","Data":"a3af5beb14ae6fe212cc4ea6188270176730dc24300dda9531023911fbd89209"} Oct 30 02:04:36.132834 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.132706 6275 scope.go:122] "RemoveContainer" containerID="df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452" Oct 30 02:04:36.143338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.143320 6275 scope.go:122] "RemoveContainer" containerID="df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452" Oct 30 02:04:36.143617 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:36.143597 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452\": container with ID starting with df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452 not found: ID does not exist" containerID="df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452" Oct 30 02:04:36.143728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.143624 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452"} err="failed to get container status \"df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452\": rpc error: code = NotFound desc = could not find container \"df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452\": container with ID starting with df4f4daeb72acb8034bd49d200e6c8d3bacb1fedf510d318fa9e8d805ffc6452 not found: ID does not exist" Oct 30 02:04:36.168606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.168571 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsg7g\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-kube-api-access-tsg7g\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.168606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.168600 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d862bc10-3875-4427-9eee-1644c5e8e116-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.168606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.168612 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d862bc10-3875-4427-9eee-1644c5e8e116-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.168855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.168620 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d862bc10-3875-4427-9eee-1644c5e8e116-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:36.217714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.217623 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6"] Oct 30 02:04:36.230159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.230091 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-24wr6"] Oct 30 02:04:36.547160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.547113 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w"] Oct 30 02:04:36.547659 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:36.547605 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" podUID="33440bce-816d-42d3-8d44-7fb5c1cd180e" containerName="maas-api" containerID="cri-o://2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2" gracePeriod=30 Oct 30 02:04:37.090436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.090403 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:04:37.141207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.141159 6275 generic.go:356] "Generic (PLEG): container finished" podID="33440bce-816d-42d3-8d44-7fb5c1cd180e" containerID="2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2" exitCode=0 Oct 30 02:04:37.141207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.141206 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" Oct 30 02:04:37.141504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.141245 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" event={"ID":"33440bce-816d-42d3-8d44-7fb5c1cd180e","Type":"ContainerDied","Data":"2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2"} Oct 30 02:04:37.141504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.141327 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w" event={"ID":"33440bce-816d-42d3-8d44-7fb5c1cd180e","Type":"ContainerDied","Data":"23fa4eea0f06cc5d5a63275371c105698f5aa5eac48ac8a7958f4463d411a4ee"} Oct 30 02:04:37.141504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.141388 6275 scope.go:122] "RemoveContainer" containerID="2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2" Oct 30 02:04:37.158171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.156593 6275 scope.go:122] "RemoveContainer" containerID="2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2" Oct 30 02:04:37.158171 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:37.156939 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2\": container with ID starting with 2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2 not found: ID does not exist" containerID="2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2" Oct 30 02:04:37.158171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.156972 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2"} err="failed to get container status \"2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2\": rpc error: code = NotFound desc = could not find container \"2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2\": container with ID starting with 2b23ad2627a30d467d4e03edb8a49f0828829cdd5a2d26c6961a2915df20d3c2 not found: ID does not exist" Oct 30 02:04:37.215410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.215328 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t\" (UniqueName: \"kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t\") pod \"33440bce-816d-42d3-8d44-7fb5c1cd180e\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " Oct 30 02:04:37.215567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.215433 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca\") pod \"33440bce-816d-42d3-8d44-7fb5c1cd180e\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " Oct 30 02:04:37.215567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.215531 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs\") pod \"33440bce-816d-42d3-8d44-7fb5c1cd180e\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " Oct 30 02:04:37.215793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.215763 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\") pod \"33440bce-816d-42d3-8d44-7fb5c1cd180e\" (UID: \"33440bce-816d-42d3-8d44-7fb5c1cd180e\") " Oct 30 02:04:37.215864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.215785 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca" pod "33440bce-816d-42d3-8d44-7fb5c1cd180e" (UID: "33440bce-816d-42d3-8d44-7fb5c1cd180e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:37.216279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.216257 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33440bce-816d-42d3-8d44-7fb5c1cd180e-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:37.217998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.217976 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t" pod "33440bce-816d-42d3-8d44-7fb5c1cd180e" (UID: "33440bce-816d-42d3-8d44-7fb5c1cd180e"). InnerVolumeSpecName "kube-api-access-4jm6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:37.218173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.218153 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls" pod "33440bce-816d-42d3-8d44-7fb5c1cd180e" (UID: "33440bce-816d-42d3-8d44-7fb5c1cd180e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:37.218222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.218174 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs" pod "33440bce-816d-42d3-8d44-7fb5c1cd180e" (UID: "33440bce-816d-42d3-8d44-7fb5c1cd180e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:37.318053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.318020 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:37.318053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.318051 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jm6t\" (UniqueName: \"kubernetes.io/projected/33440bce-816d-42d3-8d44-7fb5c1cd180e-kube-api-access-4jm6t\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:37.318053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.318061 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33440bce-816d-42d3-8d44-7fb5c1cd180e-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:37.517199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.516714 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w"] Oct 30 02:04:37.524217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.520781 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70a069-5f4c69cdfb-5ll4w"] Oct 30 02:04:37.738807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.738777 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33440bce-816d-42d3-8d44-7fb5c1cd180e" path="/var/lib/kubelet/pods/33440bce-816d-42d3-8d44-7fb5c1cd180e/volumes" Oct 30 02:04:37.739278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:37.739261 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" path="/var/lib/kubelet/pods/d862bc10-3875-4427-9eee-1644c5e8e116/volumes" Oct 30 02:04:38.251787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:38.251750 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct"] Oct 30 02:04:38.252279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:38.251978 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" containerID="cri-o://4aca65518fb2c38d3133332b59aac24a8f9510deefbf92ff9a05649809e265a2" gracePeriod=30 Oct 30 02:04:38.261457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:38.261428 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2"] Oct 30 02:04:38.261648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:38.261614 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" containerID="cri-o://3c35cf5c8f66f6ed6a66e3d1afc86a346755643a9ef217fc4e69aee8a045f716" gracePeriod=30 Oct 30 02:04:39.792881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.792817 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 02:04:39.852357 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852323 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.852534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852372 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.852534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852423 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.852534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852463 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.852534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852506 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852582 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852725 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852798 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852849 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.852903 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token\") pod \"6b68df8f-6283-4f8e-9e70-22ac355cc809\" (UID: \"6b68df8f-6283-4f8e-9e70-22ac355cc809\") " Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.853044 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.853239 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.854087 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:39.854279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.854237 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.854744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.854371 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.854744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.854395 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68df8f-6283-4f8e-9e70-22ac355cc809-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.855350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.855329 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "kube-api-access-cttl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:39.855843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.855817 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:39.856656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.856590 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:39.862185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.853159 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:39.863951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.863922 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo" pod "6b68df8f-6283-4f8e-9e70-22ac355cc809" (UID: "6b68df8f-6283-4f8e-9e70-22ac355cc809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:39.958636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.958600 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.958636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.958634 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.958855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.958644 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttl8\" (UniqueName: \"kubernetes.io/projected/6b68df8f-6283-4f8e-9e70-22ac355cc809-kube-api-access-cttl8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.958855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.958653 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.958855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.958663 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.958855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:39.958671 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68df8f-6283-4f8e-9e70-22ac355cc809-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:40.228948 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.228854 6275 generic.go:356] "Generic (PLEG): container finished" podID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerID="8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8" exitCode=0 Oct 30 02:04:40.228948 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.228908 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" Oct 30 02:04:40.228948 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.228935 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" event={"ID":"6b68df8f-6283-4f8e-9e70-22ac355cc809","Type":"ContainerDied","Data":"8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8"} Oct 30 02:04:40.229261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.228989 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7" event={"ID":"6b68df8f-6283-4f8e-9e70-22ac355cc809","Type":"ContainerDied","Data":"a6eb757750fa3f3a2b67f388e3a5260f0f68203c5239ad1252fa5f1af030add5"} Oct 30 02:04:40.229261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.229012 6275 scope.go:122] "RemoveContainer" containerID="8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8" Oct 30 02:04:40.239761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.239743 6275 scope.go:122] "RemoveContainer" containerID="a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b" Oct 30 02:04:40.248686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.248667 6275 scope.go:122] "RemoveContainer" containerID="8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8" Oct 30 02:04:40.248952 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:40.248932 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8\": container with ID starting with 8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8 not found: ID does not exist" containerID="8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8" Oct 30 02:04:40.248999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.248962 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8"} err="failed to get container status \"8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8\": rpc error: code = NotFound desc = could not find container \"8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8\": container with ID starting with 8e53737617d07a71b9d3d707fe6b5ebf24469fe0a9c15c2469d162fd6c9c3dd8 not found: ID does not exist" Oct 30 02:04:40.248999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.248978 6275 scope.go:122] "RemoveContainer" containerID="a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b" Oct 30 02:04:40.249231 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:40.249214 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b\": container with ID starting with a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b not found: ID does not exist" containerID="a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b" Oct 30 02:04:40.249280 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.249240 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b"} err="failed to get container status \"a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b\": rpc error: code = NotFound desc = could not find container \"a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b\": container with ID starting with a898ea03d269077ea879195ab73a91c74e61ac4643ddaf1d81e52a35523e302b not found: ID does not exist" Oct 30 02:04:40.282136 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.282089 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7"] Oct 30 02:04:40.288360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:40.288335 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4fcj7"] Oct 30 02:04:41.742885 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:41.742850 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" path="/var/lib/kubelet/pods/6b68df8f-6283-4f8e-9e70-22ac355cc809/volumes" Oct 30 02:04:42.151156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:42.150711 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49"] Oct 30 02:04:42.151360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:42.151157 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" containerID="cri-o://01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0" gracePeriod=30 Oct 30 02:04:43.189918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.189885 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:04:43.190370 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.189936 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:04:43.424703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.424667 6275 generic.go:356] "Generic (PLEG): container finished" podID="2c69cb42-d10c-41c5-9529-441098f555f4" containerID="4aca65518fb2c38d3133332b59aac24a8f9510deefbf92ff9a05649809e265a2" exitCode=0 Oct 30 02:04:43.424829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.424732 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" event={"ID":"2c69cb42-d10c-41c5-9529-441098f555f4","Type":"ContainerDied","Data":"4aca65518fb2c38d3133332b59aac24a8f9510deefbf92ff9a05649809e265a2"} Oct 30 02:04:43.426825 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.426801 6275 generic.go:356] "Generic (PLEG): container finished" podID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerID="3c35cf5c8f66f6ed6a66e3d1afc86a346755643a9ef217fc4e69aee8a045f716" exitCode=0 Oct 30 02:04:43.426971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.426874 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" event={"ID":"01a1531b-aeaa-4143-af50-e4b65d5f3d20","Type":"ContainerDied","Data":"3c35cf5c8f66f6ed6a66e3d1afc86a346755643a9ef217fc4e69aee8a045f716"} Oct 30 02:04:43.606522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.606495 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 02:04:43.629481 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.629460 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 02:04:43.763357 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.763328 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.763636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.763600 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.763810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.763787 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.763970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.763952 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.764102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.764088 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.764240 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.764224 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.764392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.764374 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.764581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.764564 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.764742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.764720 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.764914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.764887 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.765658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.765630 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.765780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.765706 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.765780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.765738 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.765980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.765964 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.766077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.766018 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data\") pod \"2c69cb42-d10c-41c5-9529-441098f555f4\" (UID: \"2c69cb42-d10c-41c5-9529-441098f555f4\") " Oct 30 02:04:43.766077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.766057 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.766259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.766082 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.766259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.766109 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs\") pod \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\" (UID: \"01a1531b-aeaa-4143-af50-e4b65d5f3d20\") " Oct 30 02:04:43.767042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.765238 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.767161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.767013 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.767289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.767269 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.767565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.767546 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:43.767811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.767789 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.768106 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.768088 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:43.768382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.768359 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.768679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.768658 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.768851 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.768834 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.769050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.769024 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.769677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.769622 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "kube-api-access-b2p7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:43.770290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.770264 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.771302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.771278 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:43.771389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.771284 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "kube-api-access-sdzqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:43.771389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.771281 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:43.771389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.771341 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo" pod "01a1531b-aeaa-4143-af50-e4b65d5f3d20" (UID: "01a1531b-aeaa-4143-af50-e4b65d5f3d20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:43.771493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.771365 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:43.772437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.772417 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token" pod "2c69cb42-d10c-41c5-9529-441098f555f4" (UID: "2c69cb42-d10c-41c5-9529-441098f555f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:43.868282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868248 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868278 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c69cb42-d10c-41c5-9529-441098f555f4-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868288 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868296 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868306 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868313 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868321 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzqb\" (UniqueName: \"kubernetes.io/projected/01a1531b-aeaa-4143-af50-e4b65d5f3d20-kube-api-access-sdzqb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868329 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868337 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c69cb42-d10c-41c5-9529-441098f555f4-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868345 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2p7r\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-kube-api-access-b2p7r\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868353 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868361 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868384 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868402 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c69cb42-d10c-41c5-9529-441098f555f4-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868411 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1531b-aeaa-4143-af50-e4b65d5f3d20-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868420 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1531b-aeaa-4143-af50-e4b65d5f3d20-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868429 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:43.868524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:43.868436 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c69cb42-d10c-41c5-9529-441098f555f4-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:44.434095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.434062 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" Oct 30 02:04:44.434095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.434083 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" event={"ID":"2c69cb42-d10c-41c5-9529-441098f555f4","Type":"ContainerDied","Data":"947b085dbe254a04be1c0812c94711e0c8732389bb80a8b9acd68a25a899a51c"} Oct 30 02:04:44.434628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.434185 6275 scope.go:122] "RemoveContainer" containerID="4aca65518fb2c38d3133332b59aac24a8f9510deefbf92ff9a05649809e265a2" Oct 30 02:04:44.435639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.435626 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" Oct 30 02:04:44.435724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.435622 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2" event={"ID":"01a1531b-aeaa-4143-af50-e4b65d5f3d20","Type":"ContainerDied","Data":"8a20938dd5aaa34d0e69d94d8fe57ea9308bf66bd939296c2bd51fc981abb79c"} Oct 30 02:04:44.445203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.445180 6275 scope.go:122] "RemoveContainer" containerID="3c35cf5c8f66f6ed6a66e3d1afc86a346755643a9ef217fc4e69aee8a045f716" Oct 30 02:04:44.492442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.492407 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2"] Oct 30 02:04:44.502729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.502701 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qk7z2"] Oct 30 02:04:44.526836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.526804 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct"] Oct 30 02:04:44.531059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.531031 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct"] Oct 30 02:04:44.550894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.550832 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:04:44.550998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:44.550897 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-dp4ct" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:04:45.740773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:45.740737 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" path="/var/lib/kubelet/pods/01a1531b-aeaa-4143-af50-e4b65d5f3d20/volumes" Oct 30 02:04:45.741470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:45.741448 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" path="/var/lib/kubelet/pods/2c69cb42-d10c-41c5-9529-441098f555f4/volumes" Oct 30 02:04:46.262147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:46.262101 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb"] Oct 30 02:04:46.262344 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:46.262317 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" containerID="cri-o://7cef8c2a3b68e6b8c02365106c694a8e76006c8cc87fafe7cad91ff982a7ae9f" gracePeriod=30 Oct 30 02:04:46.697097 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:46.697008 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:04:47.464563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.464506 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 02:04:47.491715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.491687 6275 generic.go:356] "Generic (PLEG): container finished" podID="19e16a09-5127-4940-b275-4f0d20250709" containerID="01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0" exitCode=0 Oct 30 02:04:47.491878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.491734 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" Oct 30 02:04:47.491878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.491782 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" event={"ID":"19e16a09-5127-4940-b275-4f0d20250709","Type":"ContainerDied","Data":"01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0"} Oct 30 02:04:47.491878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.491840 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49" event={"ID":"19e16a09-5127-4940-b275-4f0d20250709","Type":"ContainerDied","Data":"d35a36badec591e5f8429a0328e73bec89614505a4bff2884567618a582e44c7"} Oct 30 02:04:47.491878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.491864 6275 scope.go:122] "RemoveContainer" containerID="01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0" Oct 30 02:04:47.501434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.501416 6275 scope.go:122] "RemoveContainer" containerID="01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0" Oct 30 02:04:47.501713 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:47.501696 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0\": container with ID starting with 01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0 not found: ID does not exist" containerID="01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0" Oct 30 02:04:47.501777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.501720 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0"} err="failed to get container status \"01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0\": rpc error: code = NotFound desc = could not find container \"01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0\": container with ID starting with 01824fa492dfc9fb12768af59d7cbe98e3ef6046b9fa41299b32344660e7a2f0 not found: ID does not exist" Oct 30 02:04:47.547502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547467 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.547693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547562 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.547693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547595 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.547693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547628 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.547693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547683 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.548040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547827 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.548040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547914 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:47.548040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547945 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:47.548040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.547956 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.548269 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548070 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.548269 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548155 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket\") pod \"19e16a09-5127-4940-b275-4f0d20250709\" (UID: \"19e16a09-5127-4940-b275-4f0d20250709\") " Oct 30 02:04:47.548269 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548195 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:47.548404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548286 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:47.548506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548484 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:47.548754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548732 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.548856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548761 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.548856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548778 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e16a09-5127-4940-b275-4f0d20250709-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.548856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548792 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.548856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.548804 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.551163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.550869 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "kube-api-access-h5lpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:47.551163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.551154 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:47.553537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.553505 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:47.553654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.553594 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy" pod "19e16a09-5127-4940-b275-4f0d20250709" (UID: "19e16a09-5127-4940-b275-4f0d20250709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:47.649911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.649878 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e16a09-5127-4940-b275-4f0d20250709-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.649911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.649906 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.649911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.649914 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e16a09-5127-4940-b275-4f0d20250709-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.650253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.649923 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5lpq\" (UniqueName: \"kubernetes.io/projected/19e16a09-5127-4940-b275-4f0d20250709-kube-api-access-h5lpq\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:47.829698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.829664 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49"] Oct 30 02:04:47.839955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:47.839927 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-dxv49"] Oct 30 02:04:48.666160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.665827 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc"] Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667217 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667256 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667276 6275 cpu_manager.go:414] "containerMap: removing container" podUID="cf8f9e34-a798-476c-952f-a85ba336d1a2" containerName="payload-pre-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667289 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8f9e34-a798-476c-952f-a85ba336d1a2" containerName="payload-pre-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667327 6275 cpu_manager.go:414] "containerMap: removing container" podUID="33440bce-816d-42d3-8d44-7fb5c1cd180e" containerName="maas-api" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667340 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33440bce-816d-42d3-8d44-7fb5c1cd180e" containerName="maas-api" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667364 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667378 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667407 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667419 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667438 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667449 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667486 6275 cpu_manager.go:414] "containerMap: removing container" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667498 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667528 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f0752ccc-f33c-41d7-af9a-5df5defff8ab" containerName="payload-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667540 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0752ccc-f33c-41d7-af9a-5df5defff8ab" containerName="payload-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667564 6275 cpu_manager.go:414] "containerMap: removing container" podUID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" containerName="main" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667575 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" containerName="main" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667608 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d8663dd2-2782-4466-9e62-621cd7380d64" containerName="payload-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667619 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8663dd2-2782-4466-9e62-621cd7380d64" containerName="payload-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667639 6275 cpu_manager.go:414] "containerMap: removing container" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.667650 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668060 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668103 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d862bc10-3875-4427-9eee-1644c5e8e116" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668137 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e16a09-5127-4940-b275-4f0d20250709" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668149 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0752ccc-f33c-41d7-af9a-5df5defff8ab" containerName="payload-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668160 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="967bb5b5-0617-4eb8-8cd4-0543325c11a3" containerName="main" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668172 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8663dd2-2782-4466-9e62-621cd7380d64" containerName="payload-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668181 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a1531b-aeaa-4143-af50-e4b65d5f3d20" containerName="istio-proxy" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668192 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8f9e34-a798-476c-952f-a85ba336d1a2" containerName="payload-pre-processing" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668202 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="33440bce-816d-42d3-8d44-7fb5c1cd180e" containerName="maas-api" Oct 30 02:04:48.670143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.668211 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c69cb42-d10c-41c5-9529-441098f555f4" containerName="istio-proxy" Oct 30 02:04:48.680150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.678922 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.707189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.704561 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0f3cd8-dockercfg-bdlkv\"" Oct 30 02:04:48.707189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.704790 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0f3cd8\"" Oct 30 02:04:48.720952 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.720917 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc"] Oct 30 02:04:48.794656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.794613 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.794909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.794888 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8xz\" (UniqueName: \"kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.839471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.839436 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld"] Oct 30 02:04:48.840557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.840532 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b68df8f-6283-4f8e-9e70-22ac355cc809" containerName="istio-proxy" Oct 30 02:04:48.843314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.843299 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:48.864452 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.864428 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld"] Oct 30 02:04:48.896217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.896179 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8xz\" (UniqueName: \"kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.896379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.896297 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.896479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.896463 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:48.896548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.896501 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq5rw\" (UniqueName: \"kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw\") pod \"payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:48.897307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.897264 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.920959 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.920891 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8xz\" (UniqueName: \"kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:48.997808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.997762 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:48.997956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.997822 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5rw\" (UniqueName: \"kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw\") pod \"payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:48.998471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:48.998451 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:49.020174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.020117 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5rw\" (UniqueName: \"kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw\") pod \"payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:49.024942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.024923 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:49.157802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.157767 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:49.365169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.364162 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc"] Oct 30 02:04:49.468906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.468856 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld"] Oct 30 02:04:49.471938 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:04:49.471908 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce93002_2643_4955_b305_667b7b5ee6ec.slice/crio-824390c094b5c41c27cf330e60ad40819dbce9e49ec818c511fff3437113ef3b WatchSource:0}: Error finding container 824390c094b5c41c27cf330e60ad40819dbce9e49ec818c511fff3437113ef3b: Status 404 returned error can't find the container with id 824390c094b5c41c27cf330e60ad40819dbce9e49ec818c511fff3437113ef3b Oct 30 02:04:49.507967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.507938 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" event={"ID":"dce93002-2643-4955-b305-667b7b5ee6ec","Type":"ContainerStarted","Data":"824390c094b5c41c27cf330e60ad40819dbce9e49ec818c511fff3437113ef3b"} Oct 30 02:04:49.509569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.509545 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" event={"ID":"4cff8538-4e13-426a-a33f-2fbb3d6c4397","Type":"ContainerStarted","Data":"337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42"} Oct 30 02:04:49.509666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.509590 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" event={"ID":"4cff8538-4e13-426a-a33f-2fbb3d6c4397","Type":"ContainerStarted","Data":"5069f1b7c37bfdf710d983c1d6c72e8cc4c3dff58b9cbbf7bf9f9cc38e47326e"} Oct 30 02:04:49.509767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.509744 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:04:49.594793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.594728 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" podStartSLOduration=1.5947108989999998 podStartE2EDuration="1.594710899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:49.59272015 +0000 UTC m=+2117.984527794" watchObservedRunningTime="2026-10-30 02:04:49.594710899 +0000 UTC m=+2117.986518551" Oct 30 02:04:49.764070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:49.762696 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e16a09-5127-4940-b275-4f0d20250709" path="/var/lib/kubelet/pods/19e16a09-5127-4940-b275-4f0d20250709/volumes" Oct 30 02:04:50.518114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:50.518072 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" event={"ID":"dce93002-2643-4955-b305-667b7b5ee6ec","Type":"ContainerStarted","Data":"e01b39e2a0a67e75f33dc5edb1e19375c05d5f17822592fb982970c823ed3466"} Oct 30 02:04:50.518390 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:50.518363 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:04:50.559813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:50.559764 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" podStartSLOduration=2.559751724 podStartE2EDuration="2.559751724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:50.555752605 +0000 UTC m=+2118.947560238" watchObservedRunningTime="2026-10-30 02:04:50.559751724 +0000 UTC m=+2118.951559407" Oct 30 02:04:51.526982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.526947 6275 generic.go:356] "Generic (PLEG): container finished" podID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerID="7cef8c2a3b68e6b8c02365106c694a8e76006c8cc87fafe7cad91ff982a7ae9f" exitCode=0 Oct 30 02:04:51.527466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.527027 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" event={"ID":"a6319398-2ee5-4eed-baae-2e6fe7530c9d","Type":"ContainerDied","Data":"7cef8c2a3b68e6b8c02365106c694a8e76006c8cc87fafe7cad91ff982a7ae9f"} Oct 30 02:04:51.559955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.559934 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:04:51.637048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637001 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.637278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637065 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.637278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637160 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.637278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637198 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.637278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637275 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.638353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637346 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.638353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637469 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.638353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637580 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.638353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.637621 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket\") pod \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\" (UID: \"a6319398-2ee5-4eed-baae-2e6fe7530c9d\") " Oct 30 02:04:51.642532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.641999 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "kube-api-access-l9jpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.642532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.642268 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.642720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.642695 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.643109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.642988 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:51.643270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.643249 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.643482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.643437 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.644808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.644787 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:51.645903 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.645879 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.648055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.648026 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy" pod "a6319398-2ee5-4eed-baae-2e6fe7530c9d" (UID: "a6319398-2ee5-4eed-baae-2e6fe7530c9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.740175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740109 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740143 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740153 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jpp\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-kube-api-access-l9jpp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740167 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740177 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740185 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740194 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740204 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.740458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:51.740215 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6319398-2ee5-4eed-baae-2e6fe7530c9d-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:52.534654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:52.534625 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" Oct 30 02:04:52.534654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:52.534619 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb" event={"ID":"a6319398-2ee5-4eed-baae-2e6fe7530c9d","Type":"ContainerDied","Data":"76413ed55d748a6be59903dc9bdc1351a856c2c3f42b553952ffd5a6fd70dc62"} Oct 30 02:04:52.535169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:52.534723 6275 scope.go:122] "RemoveContainer" containerID="7cef8c2a3b68e6b8c02365106c694a8e76006c8cc87fafe7cad91ff982a7ae9f" Oct 30 02:04:52.593295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:52.593255 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb"] Oct 30 02:04:52.599078 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:52.599048 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-fhgfb"] Oct 30 02:04:53.793147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:53.791906 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" path="/var/lib/kubelet/pods/a6319398-2ee5-4eed-baae-2e6fe7530c9d/volumes" Oct 30 02:04:54.447761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:54.447715 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts"] Oct 30 02:04:54.447982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:54.447948 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" podUID="4f9e3e0a-d578-47c3-9500-3e74eef20ff4" containerName="maas-api" containerID="cri-o://cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6" gracePeriod=30 Oct 30 02:04:54.990708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:54.990679 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:04:55.106397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.106367 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls\") pod \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " Oct 30 02:04:55.106593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.106512 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs\") pod \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " Oct 30 02:04:55.106593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.106553 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca\") pod \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " Oct 30 02:04:55.106723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.106658 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv\" (UniqueName: \"kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv\") pod \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\" (UID: \"4f9e3e0a-d578-47c3-9500-3e74eef20ff4\") " Oct 30 02:04:55.106961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.106939 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca" pod "4f9e3e0a-d578-47c3-9500-3e74eef20ff4" (UID: "4f9e3e0a-d578-47c3-9500-3e74eef20ff4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:55.107238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.107218 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:55.108550 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.108528 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls" pod "4f9e3e0a-d578-47c3-9500-3e74eef20ff4" (UID: "4f9e3e0a-d578-47c3-9500-3e74eef20ff4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:55.108628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.108608 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs" pod "4f9e3e0a-d578-47c3-9500-3e74eef20ff4" (UID: "4f9e3e0a-d578-47c3-9500-3e74eef20ff4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:55.108694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.108680 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv" pod "4f9e3e0a-d578-47c3-9500-3e74eef20ff4" (UID: "4f9e3e0a-d578-47c3-9500-3e74eef20ff4"). InnerVolumeSpecName "kube-api-access-t7ddv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:55.208636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.208602 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:55.208636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.208630 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ddv\" (UniqueName: \"kubernetes.io/projected/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-kube-api-access-t7ddv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:55.208636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.208640 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e3e0a-d578-47c3-9500-3e74eef20ff4-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:04:55.556162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.556098 6275 generic.go:356] "Generic (PLEG): container finished" podID="4f9e3e0a-d578-47c3-9500-3e74eef20ff4" containerID="cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6" exitCode=0 Oct 30 02:04:55.556162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.556157 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" Oct 30 02:04:55.556394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.556192 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" event={"ID":"4f9e3e0a-d578-47c3-9500-3e74eef20ff4","Type":"ContainerDied","Data":"cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6"} Oct 30 02:04:55.556394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.556247 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts" event={"ID":"4f9e3e0a-d578-47c3-9500-3e74eef20ff4","Type":"ContainerDied","Data":"f6eb363aedd92cd4c99de6bf67e0ae35080ab31c39127bff4323a229aa2d986b"} Oct 30 02:04:55.556394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.556269 6275 scope.go:122] "RemoveContainer" containerID="cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6" Oct 30 02:04:55.565872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.565852 6275 scope.go:122] "RemoveContainer" containerID="cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6" Oct 30 02:04:55.566115 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:04:55.566096 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6\": container with ID starting with cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6 not found: ID does not exist" containerID="cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6" Oct 30 02:04:55.566210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.566159 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6"} err="failed to get container status \"cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6\": rpc error: code = NotFound desc = could not find container \"cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6\": container with ID starting with cdaf4c09b3ab0e09af0be0fb50e2ae81d62d59c28531b9a81017c3d2cbe177d6 not found: ID does not exist" Oct 30 02:04:55.603770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.603744 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts"] Oct 30 02:04:55.614252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.614228 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-003ec6-679c685cfc-nbgts"] Oct 30 02:04:55.750021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:04:55.749986 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9e3e0a-d578-47c3-9500-3e74eef20ff4" path="/var/lib/kubelet/pods/4f9e3e0a-d578-47c3-9500-3e74eef20ff4/volumes" Oct 30 02:05:00.559207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:00.559164 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:05:01.529336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:01.529306 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:05:03.793105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.793054 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq"] Oct 30 02:05:03.794307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.794283 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4f9e3e0a-d578-47c3-9500-3e74eef20ff4" containerName="maas-api" Oct 30 02:05:03.794514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.794478 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9e3e0a-d578-47c3-9500-3e74eef20ff4" containerName="maas-api" Oct 30 02:05:03.794632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.794563 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" Oct 30 02:05:03.794632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.794573 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" Oct 30 02:05:03.794751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.794709 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6319398-2ee5-4eed-baae-2e6fe7530c9d" containerName="istio-proxy" Oct 30 02:05:03.794751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.794729 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9e3e0a-d578-47c3-9500-3e74eef20ff4" containerName="maas-api" Oct 30 02:05:03.799927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.799910 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.817025 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.817000 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq"] Oct 30 02:05:03.824702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.824678 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-2979be\"/\"praxis-route-2979be-kserve-self-signed-certs\"" Oct 30 02:05:03.824814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.824748 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-2979be\"/\"kube-root-ca.crt\"" Oct 30 02:05:03.828610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.828588 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-2979be\"/\"default-dockercfg-r5sf8\"" Oct 30 02:05:03.828739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.828643 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-2979be\"/\"openshift-service-ca.crt\"" Oct 30 02:05:03.893975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.893943 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.894181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.894034 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.894181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.894095 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.894316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.894242 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-457qs\" (UniqueName: \"kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.894366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.894310 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.894366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.894347 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995459 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995673 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995508 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995673 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995589 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995673 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995628 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995686 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995822 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-457qs\" (UniqueName: \"kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.995900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.995865 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.996073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.996040 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.996073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.996070 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.997899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.997879 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:03.998270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:03.998253 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:04.032761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:04.032713 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-457qs\" (UniqueName: \"kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs\") pod \"praxis-route-2979be-kserve-57c6798677-8mnvq\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:04.114660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:04.114558 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:04.484639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:04.484612 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq"] Oct 30 02:05:04.487329 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:04.487301 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded97dca7_1fc7_413b_89a0_d683114bc41d.slice/crio-4d29f52c62d7696656b27d2716a5e574a9473523581acf004471d9b190c6ef98 WatchSource:0}: Error finding container 4d29f52c62d7696656b27d2716a5e574a9473523581acf004471d9b190c6ef98: Status 404 returned error can't find the container with id 4d29f52c62d7696656b27d2716a5e574a9473523581acf004471d9b190c6ef98 Oct 30 02:05:04.696464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:04.696430 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" event={"ID":"ed97dca7-1fc7-413b-89a0-d683114bc41d","Type":"ContainerStarted","Data":"2ce5335b324d54b2d08bf65a5c73e2de287af99162fc7f70cc41f0da3efc07a6"} Oct 30 02:05:04.696464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:04.696471 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" event={"ID":"ed97dca7-1fc7-413b-89a0-d683114bc41d","Type":"ContainerStarted","Data":"4d29f52c62d7696656b27d2716a5e574a9473523581acf004471d9b190c6ef98"} Oct 30 02:05:04.737701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:04.737607 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podStartSLOduration=1.737593977 podStartE2EDuration="1.737593977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:04.734407421 +0000 UTC m=+2133.126215054" watchObservedRunningTime="2026-10-30 02:05:04.737593977 +0000 UTC m=+2133.129401647" Oct 30 02:05:05.114992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:05.114963 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:07.941852 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:07.941824 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-qfc9q_e75c1729-dbe8-4486-a74d-fa6611610662/payload-processing/0.log" Oct 30 02:05:08.163268 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:08.163232 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-qfc9q_e75c1729-dbe8-4486-a74d-fa6611610662/payload-processing/0.log" Oct 30 02:05:08.413690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:08.413661 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld_dce93002-2643-4955-b305-667b7b5ee6ec/payload-processing/0.log" Oct 30 02:05:12.974314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:12.974273 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8"] Oct 30 02:05:12.974792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:12.974537 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" podUID="e181d463-7e41-4707-949a-663cf181149c" containerName="payload-processing" containerID="cri-o://a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c" gracePeriod=30 Oct 30 02:05:12.993257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:12.993225 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d"] Oct 30 02:05:12.993741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:12.993687 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" podUID="5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" containerName="payload-pre-processing" containerID="cri-o://622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699" gracePeriod=30 Oct 30 02:05:13.293184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.293159 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:05:13.325175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.325098 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:05:13.448146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.424753 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9\" (UniqueName: \"kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9\") pod \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " Oct 30 02:05:13.457144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.448465 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume\") pod \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\" (UID: \"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7\") " Oct 30 02:05:13.467148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.457559 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume\") pod \"e181d463-7e41-4707-949a-663cf181149c\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.431550 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9" pod "5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" (UID: "5a4e6f5e-9877-49a9-8fa9-60e20bba0af7"). InnerVolumeSpecName "kube-api-access-6lvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.457705 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume" pod "5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" (UID: "5a4e6f5e-9877-49a9-8fa9-60e20bba0af7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.458266 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume" pod "e181d463-7e41-4707-949a-663cf181149c" (UID: "e181d463-7e41-4707-949a-663cf181149c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.474625 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8\" (UniqueName: \"kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8\") pod \"e181d463-7e41-4707-949a-663cf181149c\" (UID: \"e181d463-7e41-4707-949a-663cf181149c\") " Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.475250 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181d463-7e41-4707-949a-663cf181149c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.475274 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvk9\" (UniqueName: \"kubernetes.io/projected/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-kube-api-access-6lvk9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.479144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.475286 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.491378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.491291 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8" pod "e181d463-7e41-4707-949a-663cf181149c" (UID: "e181d463-7e41-4707-949a-663cf181149c"). InnerVolumeSpecName "kube-api-access-rtwl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:13.576581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.576547 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwl8\" (UniqueName: \"kubernetes.io/projected/e181d463-7e41-4707-949a-663cf181149c-kube-api-access-rtwl8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.770972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.770940 6275 generic.go:356] "Generic (PLEG): container finished" podID="e181d463-7e41-4707-949a-663cf181149c" containerID="a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c" exitCode=0 Oct 30 02:05:13.771153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.770977 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" Oct 30 02:05:13.771153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.771009 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" event={"ID":"e181d463-7e41-4707-949a-663cf181149c","Type":"ContainerDied","Data":"a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c"} Oct 30 02:05:13.771153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.771066 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8" event={"ID":"e181d463-7e41-4707-949a-663cf181149c","Type":"ContainerDied","Data":"c69e75f5a78d3a6dd074eedc05beb12b72a6c286eb7510e5afe263df8b826b47"} Oct 30 02:05:13.771153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.771092 6275 scope.go:122] "RemoveContainer" containerID="a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c" Oct 30 02:05:13.772623 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.772596 6275 generic.go:356] "Generic (PLEG): container finished" podID="5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" containerID="622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699" exitCode=0 Oct 30 02:05:13.772768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.772666 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" event={"ID":"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7","Type":"ContainerDied","Data":"622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699"} Oct 30 02:05:13.772768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.772711 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" event={"ID":"5a4e6f5e-9877-49a9-8fa9-60e20bba0af7","Type":"ContainerDied","Data":"2ea6bea41d5e59488de4ca937f0b83e66f72b5c44aa69d432b2c425e594bda63"} Oct 30 02:05:13.772768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.772676 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d" Oct 30 02:05:13.822587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.817227 6275 scope.go:122] "RemoveContainer" containerID="a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c" Oct 30 02:05:13.836186 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:05:13.829453 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c\": container with ID starting with a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c not found: ID does not exist" containerID="a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c" Oct 30 02:05:13.836186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.829512 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c"} err="failed to get container status \"a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c\": rpc error: code = NotFound desc = could not find container \"a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c\": container with ID starting with a9059d6d29019b4e3084b9a938c3f410d32d2574cae5cb046afeb7d6f994cf4c not found: ID does not exist" Oct 30 02:05:13.836186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.829541 6275 scope.go:122] "RemoveContainer" containerID="622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699" Oct 30 02:05:13.836186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.834559 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d"] Oct 30 02:05:13.844247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.844204 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c90f5-d4cb4bf67-zzn9d"] Oct 30 02:05:13.856300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.856198 6275 scope.go:122] "RemoveContainer" containerID="622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699" Oct 30 02:05:13.856872 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:05:13.856799 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699\": container with ID starting with 622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699 not found: ID does not exist" containerID="622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699" Oct 30 02:05:13.856872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.856836 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699"} err="failed to get container status \"622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699\": rpc error: code = NotFound desc = could not find container \"622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699\": container with ID starting with 622f8d9985c1cc66781284a1c160bdf7ef540c74f86ce8b5b334f4591f0c9699 not found: ID does not exist" Oct 30 02:05:13.865552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.865521 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8"] Oct 30 02:05:13.870130 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:13.870094 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-74b8894498-hx8v8"] Oct 30 02:05:14.115781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:14.115753 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:14.129711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:14.129680 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:14.793074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:14.793039 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:05:15.739581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:15.739546 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" path="/var/lib/kubelet/pods/5a4e6f5e-9877-49a9-8fa9-60e20bba0af7/volumes" Oct 30 02:05:15.740082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:15.740064 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e181d463-7e41-4707-949a-663cf181149c" path="/var/lib/kubelet/pods/e181d463-7e41-4707-949a-663cf181149c/volumes" Oct 30 02:05:16.696875 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:05:16.696841 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:05:28.156706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.156664 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht"] Oct 30 02:05:28.157500 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.157477 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e181d463-7e41-4707-949a-663cf181149c" containerName="payload-processing" Oct 30 02:05:28.157500 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.157500 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e181d463-7e41-4707-949a-663cf181149c" containerName="payload-processing" Oct 30 02:05:28.157672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.157524 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" containerName="payload-pre-processing" Oct 30 02:05:28.157672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.157532 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" containerName="payload-pre-processing" Oct 30 02:05:28.157776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.157733 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e181d463-7e41-4707-949a-663cf181149c" containerName="payload-processing" Oct 30 02:05:28.157776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.157749 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4e6f5e-9877-49a9-8fa9-60e20bba0af7" containerName="payload-pre-processing" Oct 30 02:05:28.160685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.160663 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.171501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.171479 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5"] Oct 30 02:05:28.176108 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.176089 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc"] Oct 30 02:05:28.176247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.176220 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.179954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.179908 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.181785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.181694 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-78d5ff-openshift-default-dockercfg-s66rj\"" Oct 30 02:05:28.187554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.187509 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht"] Oct 30 02:05:28.202273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.202250 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5"] Oct 30 02:05:28.208113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.208091 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc"] Oct 30 02:05:28.290379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290346 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.290558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290394 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llc22\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.290558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290468 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.290558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290528 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.290558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290551 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.290792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290668 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.290792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290701 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.290792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290730 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.290792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290755 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.290934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290791 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.290934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290829 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.290934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290861 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.291029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290941 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.291029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.290988 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.291089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291062 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7tr\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.291089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291082 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvvn\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.291190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291149 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.291299 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291238 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.291393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291302 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.291393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291322 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.291393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291375 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.291553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291446 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.291553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291472 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.291553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291491 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.291553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291521 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.291553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291545 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.291745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.291564 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.392525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392496 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.392525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392528 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.392753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392548 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.392753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392565 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.392753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392593 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.392753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392721 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.392965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392779 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.392965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392826 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.392965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392873 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.392965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392917 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7tr\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.392965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.392946 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvvn\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.393261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393015 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.393261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393074 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.393496 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393463 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.393570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393554 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.393630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393002 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.393630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393601 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.393630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393615 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.393783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393706 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.393783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393753 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.393901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393804 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.393901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393832 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.393901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393881 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.394058 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393921 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.394393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.394366 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.394507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.394433 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.394507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393026 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.394713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.394686 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.394962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.393056 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.395041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.395025 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.395221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.395200 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.395652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.395627 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.396013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.395863 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.396013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.395978 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.396176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396035 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.396176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396076 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llc22\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.396176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396110 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.396330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396256 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.396330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396288 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.396330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396310 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.396892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396864 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.397017 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.396996 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.397695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.397673 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.397906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.397888 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.400262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.400242 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.401816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.400749 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.401816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.401076 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.402549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.402523 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.418644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.418274 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.418644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.418424 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.418644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.418443 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.418644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.418589 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvvn\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.418644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.418589 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7tr\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.418875 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.418794 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llc22\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.478020 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.477991 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:28.495616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.495595 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:28.506230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.506202 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:28.803876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.803852 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht"] Oct 30 02:05:28.805909 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:28.805879 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a6b4ea_7cae_4540_8ca8_ac9912fd70ee.slice/crio-87d822d93d97ab6dce7df5f777c28a1097ddb98a86ea44c16b5ac8f878dbe625 WatchSource:0}: Error finding container 87d822d93d97ab6dce7df5f777c28a1097ddb98a86ea44c16b5ac8f878dbe625: Status 404 returned error can't find the container with id 87d822d93d97ab6dce7df5f777c28a1097ddb98a86ea44c16b5ac8f878dbe625 Oct 30 02:05:28.808284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.808248 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.808367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.808313 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.808367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.808344 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.828917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.828886 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5"] Oct 30 02:05:28.831613 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:28.831589 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052c0793_2f29_41d8_8afa_9ca36b98789e.slice/crio-f915f48682bd3c821fdc03c944a818e716a40cc010a43d85c74ccb4eb5ebf72e WatchSource:0}: Error finding container f915f48682bd3c821fdc03c944a818e716a40cc010a43d85c74ccb4eb5ebf72e: Status 404 returned error can't find the container with id f915f48682bd3c821fdc03c944a818e716a40cc010a43d85c74ccb4eb5ebf72e Oct 30 02:05:28.834101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.834067 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.834228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.834207 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.834297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.834258 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.844017 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.843990 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc"] Oct 30 02:05:28.846780 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:28.846753 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96fb179_75e5_443d_ae05_c4c42e126abd.slice/crio-e179754ad67a90f82cafa5a45c5cfc638d77fef6520445e9fe2604acebf7617c WatchSource:0}: Error finding container e179754ad67a90f82cafa5a45c5cfc638d77fef6520445e9fe2604acebf7617c: Status 404 returned error can't find the container with id e179754ad67a90f82cafa5a45c5cfc638d77fef6520445e9fe2604acebf7617c Oct 30 02:05:28.850378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.850337 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.850487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.850419 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.850487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.850460 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:28.968983 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.968833 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" event={"ID":"f96fb179-75e5-443d-ae05-c4c42e126abd","Type":"ContainerStarted","Data":"268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef"} Oct 30 02:05:28.968983 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.968905 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" event={"ID":"f96fb179-75e5-443d-ae05-c4c42e126abd","Type":"ContainerStarted","Data":"e179754ad67a90f82cafa5a45c5cfc638d77fef6520445e9fe2604acebf7617c"} Oct 30 02:05:29.007158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.995223 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" event={"ID":"052c0793-2f29-41d8-8afa-9ca36b98789e","Type":"ContainerStarted","Data":"fe9c2de58d4c10024b83bf283aee34a976d124c3a4fd9838be6ad8fe31262083"} Oct 30 02:05:29.007158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:28.995290 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" event={"ID":"052c0793-2f29-41d8-8afa-9ca36b98789e","Type":"ContainerStarted","Data":"f915f48682bd3c821fdc03c944a818e716a40cc010a43d85c74ccb4eb5ebf72e"} Oct 30 02:05:29.015770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.014664 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" event={"ID":"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee","Type":"ContainerStarted","Data":"211e79c73180356922536452775ec9b97d3f59588434f921d4d5e81337eb636f"} Oct 30 02:05:29.015770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.014807 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" event={"ID":"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee","Type":"ContainerStarted","Data":"87d822d93d97ab6dce7df5f777c28a1097ddb98a86ea44c16b5ac8f878dbe625"} Oct 30 02:05:29.090454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.090342 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podStartSLOduration=1.090322478 podStartE2EDuration="1.090322478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:29.088560781 +0000 UTC m=+2157.480368414" watchObservedRunningTime="2026-10-30 02:05:29.090322478 +0000 UTC m=+2157.482130112" Oct 30 02:05:29.091865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.091809 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podStartSLOduration=1.091794337 podStartE2EDuration="1.091794337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:29.045490477 +0000 UTC m=+2157.437298123" watchObservedRunningTime="2026-10-30 02:05:29.091794337 +0000 UTC m=+2157.483601968" Oct 30 02:05:29.134221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.134153 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podStartSLOduration=1.134104278 podStartE2EDuration="1.134104278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:29.131216171 +0000 UTC m=+2157.523023804" watchObservedRunningTime="2026-10-30 02:05:29.134104278 +0000 UTC m=+2157.525911913" Oct 30 02:05:29.493893 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.485807 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:29.494425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.493910 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:29.494425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.493971 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:29.503729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.500518 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:29.508177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.504693 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:29.508177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.504757 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:29.510161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.510111 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:29.513156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.512331 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:29.513156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:29.512419 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:30.479554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.479516 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:30.479952 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.479918 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:30.501408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.501370 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:30.502006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.501978 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:30.509611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.509494 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:30.509611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.509553 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:30.922976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.922935 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt"] Oct 30 02:05:30.934504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:30.934460 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035555 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035616 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035653 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035723 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035751 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035790 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.035813 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhcr2\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.036081 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.036278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.036153 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.136162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.136100 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt"] Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137097 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137198 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137269 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137313 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137350 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137410 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137447 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhcr2\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137588 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137635 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.137990 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.138062 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.139220 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.186366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.139879 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.326417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.275767 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.326417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.297667 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.337793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.337760 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhcr2\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.428073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.426903 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.431461 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.431429 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:31.483714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.483179 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:31.483714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.483241 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:31.496610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.496495 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:31.496610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.496553 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:31.508954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.508260 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:31.508954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.508320 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:31.654571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:31.654529 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:32.517204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.483624 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:32.517204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.483684 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:32.517204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.498605 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:32.517204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.498683 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:32.517204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.509114 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:32.517204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.509182 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:32.615744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.599748 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt"] Oct 30 02:05:32.622289 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:32.620253 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode091fd66_e4b9_40d6_8687_5f4d78ea782b.slice/crio-804ed95dc565c93482ab40f2629989b0d9b4b1d0deff6f49e4399246150615b6 WatchSource:0}: Error finding container 804ed95dc565c93482ab40f2629989b0d9b4b1d0deff6f49e4399246150615b6: Status 404 returned error can't find the container with id 804ed95dc565c93482ab40f2629989b0d9b4b1d0deff6f49e4399246150615b6 Oct 30 02:05:32.623641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.623602 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:32.623748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.623694 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:32.623748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:32.623742 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:33.469867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.469830 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" event={"ID":"e091fd66-e4b9-40d6-8687-5f4d78ea782b","Type":"ContainerStarted","Data":"6f375b1d8630658f9129f7cf77e822598686a066d4de6b202f39529ab619b66c"} Oct 30 02:05:33.470216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.470180 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" event={"ID":"e091fd66-e4b9-40d6-8687-5f4d78ea782b","Type":"ContainerStarted","Data":"804ed95dc565c93482ab40f2629989b0d9b4b1d0deff6f49e4399246150615b6"} Oct 30 02:05:33.479596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.479566 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.479814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.479788 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:33.502147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.500784 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.502147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.500838 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:33.514147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.510481 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.514147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.510534 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:33.533342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.533277 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podStartSLOduration=3.5332566549999997 podStartE2EDuration="3.533256655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:33.528794584 +0000 UTC m=+2161.920602223" watchObservedRunningTime="2026-10-30 02:05:33.533256655 +0000 UTC m=+2161.925064291" Oct 30 02:05:33.669146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.656642 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:33.691146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.673294 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.691146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.673374 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:33.835150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:33.830082 6275 scope.go:122] "RemoveContainer" containerID="35cc0d51c9032838d43a63bb3cbe9a0fc27e6e06575df138e2719bd7061fba86" Oct 30 02:05:34.486953 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.486513 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.486953 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.486575 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:34.497064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.496942 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.497064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.497003 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:34.513489 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.513455 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.513784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.513754 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:34.640336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.640223 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:34.656587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.656550 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.656868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.656839 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:34.760146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.755220 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:34.784638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:34.784589 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:35.122144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.121764 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:35.482236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.481259 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.482236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.481327 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:35.497336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.496703 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.497336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.496765 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:35.507039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.506650 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.507039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.506705 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:35.657148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.656709 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.657148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:35.656788 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:36.411148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.410581 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v"] Oct 30 02:05:36.412505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.411379 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" podUID="3a043753-45b2-4147-b917-ef00e11ce6d8" containerName="maas-api" containerID="cri-o://c6e858d98dc8a7139a53ba018a858a05829c057114de6aacf2b3b5adeeec3922" gracePeriod=30 Oct 30 02:05:36.526622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.493310 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.527015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.526981 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 02:05:36.527225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.513916 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.527409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.527375 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:05:36.527919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.526419 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.528092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.528064 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 02:05:36.623908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.623866 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf"] Oct 30 02:05:36.624427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.624372 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" podUID="aebe6051-c40d-4051-9349-734c4185cbf4" containerName="payload-processing" containerID="cri-o://8163ce6f11e600217b36877c73ecb0d996b7d70ce94a7d2d29af39e8af8ed164" gracePeriod=30 Oct 30 02:05:36.753220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.752251 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.753220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:36.752309 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:37.605975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.605776 6275 generic.go:356] "Generic (PLEG): container finished" podID="aebe6051-c40d-4051-9349-734c4185cbf4" containerID="8163ce6f11e600217b36877c73ecb0d996b7d70ce94a7d2d29af39e8af8ed164" exitCode=0 Oct 30 02:05:37.605975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.605891 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" event={"ID":"aebe6051-c40d-4051-9349-734c4185cbf4","Type":"ContainerDied","Data":"8163ce6f11e600217b36877c73ecb0d996b7d70ce94a7d2d29af39e8af8ed164"} Oct 30 02:05:37.757148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.753217 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:37.803177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.792504 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": context deadline exceeded" Oct 30 02:05:37.803177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.793065 6275 generic.go:356] "Generic (PLEG): container finished" podID="3a043753-45b2-4147-b917-ef00e11ce6d8" containerID="c6e858d98dc8a7139a53ba018a858a05829c057114de6aacf2b3b5adeeec3922" exitCode=0 Oct 30 02:05:37.803177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.793232 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" event={"ID":"3a043753-45b2-4147-b917-ef00e11ce6d8","Type":"ContainerDied","Data":"c6e858d98dc8a7139a53ba018a858a05829c057114de6aacf2b3b5adeeec3922"} Oct 30 02:05:37.803177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.757412 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:37.862876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.862839 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:37.863255 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.863104 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:37.909900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.909860 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:37.913049 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.912257 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:37.933964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.933352 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:05:37.971199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.968439 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:37.974953 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.974921 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:37.992560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.991621 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:37.992560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.992214 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:37.993016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:37.992996 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:05:38.007612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.007367 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:05:38.014382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.014323 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:05:38.108687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.108604 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.219762 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls\") pod \"3a043753-45b2-4147-b917-ef00e11ce6d8\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.219841 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs\") pod \"3a043753-45b2-4147-b917-ef00e11ce6d8\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.219933 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume\") pod \"aebe6051-c40d-4051-9349-734c4185cbf4\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.220216 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp\" (UniqueName: \"kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp\") pod \"3a043753-45b2-4147-b917-ef00e11ce6d8\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.220336 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2\" (UniqueName: \"kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2\") pod \"aebe6051-c40d-4051-9349-734c4185cbf4\" (UID: \"aebe6051-c40d-4051-9349-734c4185cbf4\") " Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.220379 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca\") pod \"3a043753-45b2-4147-b917-ef00e11ce6d8\" (UID: \"3a043753-45b2-4147-b917-ef00e11ce6d8\") " Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.221730 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume" pod "aebe6051-c40d-4051-9349-734c4185cbf4" (UID: "aebe6051-c40d-4051-9349-734c4185cbf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:38.223306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.222110 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca" pod "3a043753-45b2-4147-b917-ef00e11ce6d8" (UID: "3a043753-45b2-4147-b917-ef00e11ce6d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:38.227941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.227878 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2" pod "aebe6051-c40d-4051-9349-734c4185cbf4" (UID: "aebe6051-c40d-4051-9349-734c4185cbf4"). InnerVolumeSpecName "kube-api-access-r44l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:38.228552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.228487 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp" pod "3a043753-45b2-4147-b917-ef00e11ce6d8" (UID: "3a043753-45b2-4147-b917-ef00e11ce6d8"). InnerVolumeSpecName "kube-api-access-shmgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:38.231613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.231484 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls" pod "3a043753-45b2-4147-b917-ef00e11ce6d8" (UID: "3a043753-45b2-4147-b917-ef00e11ce6d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:38.231613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.231579 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs" pod "3a043753-45b2-4147-b917-ef00e11ce6d8" (UID: "3a043753-45b2-4147-b917-ef00e11ce6d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:38.324391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.322707 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shmgp\" (UniqueName: \"kubernetes.io/projected/3a043753-45b2-4147-b917-ef00e11ce6d8-kube-api-access-shmgp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.324391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.322765 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r44l2\" (UniqueName: \"kubernetes.io/projected/aebe6051-c40d-4051-9349-734c4185cbf4-kube-api-access-r44l2\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.324391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.322781 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a043753-45b2-4147-b917-ef00e11ce6d8-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.324391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.322796 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.324391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.322828 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a043753-45b2-4147-b917-ef00e11ce6d8-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.324391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.322841 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebe6051-c40d-4051-9349-734c4185cbf4-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.666166 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.666099 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:38.666386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.666182 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:38.801179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.801148 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" Oct 30 02:05:38.801179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.801157 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf" event={"ID":"aebe6051-c40d-4051-9349-734c4185cbf4","Type":"ContainerDied","Data":"deb9716ab3ade2822cecdc190d2970eada9b8883d5cc598ff7a2903ed9eee3a2"} Oct 30 02:05:38.801724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.801238 6275 scope.go:122] "RemoveContainer" containerID="8163ce6f11e600217b36877c73ecb0d996b7d70ce94a7d2d29af39e8af8ed164" Oct 30 02:05:38.802804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.802787 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" Oct 30 02:05:38.802931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.802803 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" event={"ID":"3a043753-45b2-4147-b917-ef00e11ce6d8","Type":"ContainerDied","Data":"67728531b3f7d02402c0ea760f79399304a87fa2b474d3430f5546e9140367c6"} Oct 30 02:05:38.812432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.812416 6275 scope.go:122] "RemoveContainer" containerID="c6e858d98dc8a7139a53ba018a858a05829c057114de6aacf2b3b5adeeec3922" Oct 30 02:05:38.867326 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.867292 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf"] Oct 30 02:05:38.871736 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.871707 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-7495c56bd-gpdbf"] Oct 30 02:05:38.895198 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.895167 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v"] Oct 30 02:05:38.898355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:38.898322 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v"] Oct 30 02:05:39.666829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:39.666788 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:39.667175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:39.667114 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:39.762473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:39.762432 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a043753-45b2-4147-b917-ef00e11ce6d8" path="/var/lib/kubelet/pods/3a043753-45b2-4147-b917-ef00e11ce6d8/volumes" Oct 30 02:05:39.763438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:39.763393 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aebe6051-c40d-4051-9349-734c4185cbf4" path="/var/lib/kubelet/pods/aebe6051-c40d-4051-9349-734c4185cbf4/volumes" Oct 30 02:05:40.655260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:40.655219 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:05:40.655655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:40.655283 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 02:05:40.929184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:40.929036 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-168989-86c6d6766d-m5j9v" podUID="3a043753-45b2-4147-b917-ef00e11ce6d8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.150:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:41.659956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:41.659926 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:41.660375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:41.660213 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:41.660934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:41.660917 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:05:42.713512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:42.713468 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742"] Oct 30 02:05:42.714068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:42.713844 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" podUID="9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" containerName="payload-processing" containerID="cri-o://a156b9b321899a75bca5e3326313c3785ced26159f4fb37c581ccba5d4a641fe" gracePeriod=30 Oct 30 02:05:42.726677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:42.726645 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j"] Oct 30 02:05:42.726857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:42.726836 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" podUID="811316d1-343e-44b8-86f9-2a100937d91d" containerName="payload-pre-processing" containerID="cri-o://1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c" gracePeriod=30 Oct 30 02:05:42.833190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:42.832907 6275 generic.go:356] "Generic (PLEG): container finished" podID="9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" containerID="a156b9b321899a75bca5e3326313c3785ced26159f4fb37c581ccba5d4a641fe" exitCode=0 Oct 30 02:05:42.833190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:42.832976 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" event={"ID":"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f","Type":"ContainerDied","Data":"a156b9b321899a75bca5e3326313c3785ced26159f4fb37c581ccba5d4a641fe"} Oct 30 02:05:43.031771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.031745 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:05:43.035112 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.035090 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:05:43.120154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.119779 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume\") pod \"811316d1-343e-44b8-86f9-2a100937d91d\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " Oct 30 02:05:43.120154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.119917 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s\" (UniqueName: \"kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s\") pod \"811316d1-343e-44b8-86f9-2a100937d91d\" (UID: \"811316d1-343e-44b8-86f9-2a100937d91d\") " Oct 30 02:05:43.120154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.119998 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume\") pod \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " Oct 30 02:05:43.120154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.120161 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5\" (UniqueName: \"kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5\") pod \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\" (UID: \"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f\") " Oct 30 02:05:43.140165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.133695 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume" pod "811316d1-343e-44b8-86f9-2a100937d91d" (UID: "811316d1-343e-44b8-86f9-2a100937d91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:43.171796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.160381 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume" pod "9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" (UID: "9022ffe1-ea98-4fc1-b4ad-344020ff4a3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.211438 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5" pod "9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" (UID: "9022ffe1-ea98-4fc1-b4ad-344020ff4a3f"). InnerVolumeSpecName "kube-api-access-n59s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.211957 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s" pod "811316d1-343e-44b8-86f9-2a100937d91d" (UID: "811316d1-343e-44b8-86f9-2a100937d91d"). InnerVolumeSpecName "kube-api-access-ffp2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.231137 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4"] Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232104 6275 cpu_manager.go:414] "containerMap: removing container" podUID="811316d1-343e-44b8-86f9-2a100937d91d" containerName="payload-pre-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232143 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811316d1-343e-44b8-86f9-2a100937d91d" containerName="payload-pre-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232167 6275 cpu_manager.go:414] "containerMap: removing container" podUID="aebe6051-c40d-4051-9349-734c4185cbf4" containerName="payload-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232178 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aebe6051-c40d-4051-9349-734c4185cbf4" containerName="payload-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232236 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" containerName="payload-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232243 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" containerName="payload-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232267 6275 cpu_manager.go:414] "containerMap: removing container" podUID="3a043753-45b2-4147-b917-ef00e11ce6d8" containerName="maas-api" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232274 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a043753-45b2-4147-b917-ef00e11ce6d8" containerName="maas-api" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232420 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" containerName="payload-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232435 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a043753-45b2-4147-b917-ef00e11ce6d8" containerName="maas-api" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232448 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="811316d1-343e-44b8-86f9-2a100937d91d" containerName="payload-pre-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.232459 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="aebe6051-c40d-4051-9349-734c4185cbf4" containerName="payload-processing" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.235821 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.235868 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59s5\" (UniqueName: \"kubernetes.io/projected/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f-kube-api-access-n59s5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.235891 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811316d1-343e-44b8-86f9-2a100937d91d-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.249563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.235906 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffp2s\" (UniqueName: \"kubernetes.io/projected/811316d1-343e-44b8-86f9-2a100937d91d-kube-api-access-ffp2s\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.274506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.274473 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.293263 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.281430 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4"] Oct 30 02:05:43.337327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337290 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337396 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337440 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p2w\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337566 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337650 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337734 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337777 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337804 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.337973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.337937 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.438790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.438748 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.438806 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.438857 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.438894 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439023 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439182 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439252 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439269 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439305 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439306 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439344 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439397 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p2w\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439553 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.439948 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.439926 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.441330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.441300 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.441601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.441584 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.460479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.460455 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.460589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.460570 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p2w\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.611521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.611474 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:43.844045 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.844009 6275 generic.go:356] "Generic (PLEG): container finished" podID="811316d1-343e-44b8-86f9-2a100937d91d" containerID="1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c" exitCode=0 Oct 30 02:05:43.844502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.844059 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" Oct 30 02:05:43.844502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.844097 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" event={"ID":"811316d1-343e-44b8-86f9-2a100937d91d","Type":"ContainerDied","Data":"1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c"} Oct 30 02:05:43.844502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.844191 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j" event={"ID":"811316d1-343e-44b8-86f9-2a100937d91d","Type":"ContainerDied","Data":"354403d92585362cafb0ca37defc768202cb26b423a616be66a9f82767da4d6a"} Oct 30 02:05:43.844502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.844223 6275 scope.go:122] "RemoveContainer" containerID="1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c" Oct 30 02:05:43.845661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.845639 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" Oct 30 02:05:43.845805 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.845633 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742" event={"ID":"9022ffe1-ea98-4fc1-b4ad-344020ff4a3f","Type":"ContainerDied","Data":"180ca05b746e5cd173fe47d4ca490f9d1c5e194acdf27e1b03b173b9dc9095a4"} Oct 30 02:05:43.854231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.854213 6275 scope.go:122] "RemoveContainer" containerID="1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c" Oct 30 02:05:43.854503 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:05:43.854485 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c\": container with ID starting with 1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c not found: ID does not exist" containerID="1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c" Oct 30 02:05:43.854547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.854511 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c"} err="failed to get container status \"1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c\": rpc error: code = NotFound desc = could not find container \"1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c\": container with ID starting with 1a7c3aa9c372f2edcb22bb278fef726d8a59737dded4c3663a6cfc4f5fa4f90c not found: ID does not exist" Oct 30 02:05:43.854547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.854527 6275 scope.go:122] "RemoveContainer" containerID="a156b9b321899a75bca5e3326313c3785ced26159f4fb37c581ccba5d4a641fe" Oct 30 02:05:43.891946 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.891917 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j"] Oct 30 02:05:43.908035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.908006 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-145336-755cc54c67-gdp9j"] Oct 30 02:05:43.937686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.937661 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742"] Oct 30 02:05:43.941534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.941511 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-7dbd64fd96-hb742"] Oct 30 02:05:43.950384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.950344 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4"] Oct 30 02:05:43.953477 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:43.953438 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b61c35_b18f_443a_aa16_ff4678888a49.slice/crio-5b69938da55459e9838c4b055ee9e858a33b8e7cc2282c08e8d97702190df373 WatchSource:0}: Error finding container 5b69938da55459e9838c4b055ee9e858a33b8e7cc2282c08e8d97702190df373: Status 404 returned error can't find the container with id 5b69938da55459e9838c4b055ee9e858a33b8e7cc2282c08e8d97702190df373 Oct 30 02:05:43.955887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.955855 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:43.955978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.955922 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:43.955978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:43.955961 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:44.872258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:44.872218 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" event={"ID":"58b61c35-b18f-443a-aa16-ff4678888a49","Type":"ContainerStarted","Data":"d21e6b4853be94efdb0f2a00428a384203a6bb155cb84dcb0c469283dacf0f85"} Oct 30 02:05:44.872721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:44.872278 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" event={"ID":"58b61c35-b18f-443a-aa16-ff4678888a49","Type":"ContainerStarted","Data":"5b69938da55459e9838c4b055ee9e858a33b8e7cc2282c08e8d97702190df373"} Oct 30 02:05:44.962330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:44.962256 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podStartSLOduration=1.962231342 podStartE2EDuration="1.962231342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:44.932735834 +0000 UTC m=+2173.324543470" watchObservedRunningTime="2026-10-30 02:05:44.962231342 +0000 UTC m=+2173.354038976" Oct 30 02:05:45.229098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.229005 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr"] Oct 30 02:05:45.229316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.229278 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" containerID="cri-o://04a1e312d91e3b59d2714c1998c0b3ae7a5a8a3fa3f2948fc7c76795b5011932" gracePeriod=30 Oct 30 02:05:45.269666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.269618 6275 patch_prober.go:29] interesting pod/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:05:45.269837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.269679 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:05:45.615877 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.613213 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:45.619115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.619084 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:45.619403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.619379 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:45.747147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.741292 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811316d1-343e-44b8-86f9-2a100937d91d" path="/var/lib/kubelet/pods/811316d1-343e-44b8-86f9-2a100937d91d/volumes" Oct 30 02:05:45.747147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:45.741923 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9022ffe1-ea98-4fc1-b4ad-344020ff4a3f" path="/var/lib/kubelet/pods/9022ffe1-ea98-4fc1-b4ad-344020ff4a3f/volumes" Oct 30 02:05:46.612448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:46.612413 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:46.612952 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:46.612485 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:46.696363 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:05:46.696330 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:05:47.612693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:47.612656 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:47.613209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:47.612716 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:48.612336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:48.612299 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:48.612529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:48.612364 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:49.612745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:49.612703 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:49.613172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:49.612772 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:50.624718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.624270 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:50.624718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.624333 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:50.959314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.958626 6275 generic.go:356] "Generic (PLEG): container finished" podID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerID="04a1e312d91e3b59d2714c1998c0b3ae7a5a8a3fa3f2948fc7c76795b5011932" exitCode=0 Oct 30 02:05:50.959314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.958679 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" event={"ID":"cf4979a4-b725-4497-bc66-6c6779fbcff1","Type":"ContainerDied","Data":"04a1e312d91e3b59d2714c1998c0b3ae7a5a8a3fa3f2948fc7c76795b5011932"} Oct 30 02:05:50.959314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.958734 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" event={"ID":"cf4979a4-b725-4497-bc66-6c6779fbcff1","Type":"ContainerDied","Data":"7562e937ddd3deeb1143bb2a801c0f485a163cd087e948935e0fc47d1a6abba3"} Oct 30 02:05:50.959314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.958760 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7562e937ddd3deeb1143bb2a801c0f485a163cd087e948935e0fc47d1a6abba3" Oct 30 02:05:50.959314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:50.958779 6275 scope.go:122] "RemoveContainer" containerID="398e81107bc9832c90de9435b6b8d0bf35c0859f9b58cfe1d2159a17d689edb2" Oct 30 02:05:51.002579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.002547 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:05:51.036725 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.036680 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.036881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.036763 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.036881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.036827 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.036940 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.037000 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.037057 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.037183 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.037234 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.037276 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs\") pod \"cf4979a4-b725-4497-bc66-6c6779fbcff1\" (UID: \"cf4979a4-b725-4497-bc66-6c6779fbcff1\") " Oct 30 02:05:51.037874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.037832 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:51.038249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.038047 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:51.038437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.038401 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:51.041374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.041345 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:51.042854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.042831 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:05:51.044028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.044005 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:51.046620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.046584 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "kube-api-access-hfpr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:51.046972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.046937 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:51.048872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.048796 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token" pod "cf4979a4-b725-4497-bc66-6c6779fbcff1" (UID: "cf4979a4-b725-4497-bc66-6c6779fbcff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213810 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213851 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213866 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213879 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4979a4-b725-4497-bc66-6c6779fbcff1-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213893 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfpr8\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-kube-api-access-hfpr8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213907 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213920 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213932 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4979a4-b725-4497-bc66-6c6779fbcff1-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.214030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.213945 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4979a4-b725-4497-bc66-6c6779fbcff1-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:05:51.616145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.613455 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:05:51.616145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.613534 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 02:05:51.987565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:51.987477 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr" Oct 30 02:05:52.137145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:52.136387 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr"] Oct 30 02:05:52.158829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:52.158788 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-168989-openshift-default-6c7887fd69-5g8qr"] Oct 30 02:05:52.762739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:52.761112 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:52.762739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:52.762593 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:52.763891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:52.763871 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:05:53.742589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:53.742544 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" path="/var/lib/kubelet/pods/cf4979a4-b725-4497-bc66-6c6779fbcff1/volumes" Oct 30 02:05:55.339938 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.339896 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6"] Oct 30 02:05:55.341429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.341397 6275 cpu_manager.go:414] "containerMap: removing container" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" Oct 30 02:05:55.341429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.341429 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" Oct 30 02:05:55.341594 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.341472 6275 cpu_manager.go:414] "containerMap: removing container" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" Oct 30 02:05:55.341594 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.341480 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" Oct 30 02:05:55.341674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.341666 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" Oct 30 02:05:55.341709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.341682 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4979a4-b725-4497-bc66-6c6779fbcff1" containerName="istio-proxy" Oct 30 02:05:55.345317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.345298 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358226 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358268 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358368 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358426 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358510 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358557 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7ll\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358611 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358697 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358684 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.358737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.358714 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.374739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.374705 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6"] Oct 30 02:05:55.381801 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.381771 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-428529-openshift-default-dockercfg-swxvs\"" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498289 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498389 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498430 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498457 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498565 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498638 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7ll\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498716 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498784 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.501148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.498824 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.502317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.502288 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.529146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.505764 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.529146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.506458 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.529146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.506712 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.529146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.506990 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.557550 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.557512 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.558086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.558053 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.558889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.558866 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.597196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.581310 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7ll\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll\") pod \"e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:55.667445 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:55.667384 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:56.032700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:56.032518 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6"] Oct 30 02:05:56.035238 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:05:56.035210 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e030d_cd33_4bca_9cdc_f72c4818275b.slice/crio-a4cfb33a37cb80c5e0ca9e04cd5aa1c565e43998241d8489eaceb7f8cefb1735 WatchSource:0}: Error finding container a4cfb33a37cb80c5e0ca9e04cd5aa1c565e43998241d8489eaceb7f8cefb1735: Status 404 returned error can't find the container with id a4cfb33a37cb80c5e0ca9e04cd5aa1c565e43998241d8489eaceb7f8cefb1735 Oct 30 02:05:56.037586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:56.037554 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:56.037653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:56.037620 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:56.037701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:56.037658 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:05:57.027613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:57.027577 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" event={"ID":"e41e030d-cd33-4bca-9cdc-f72c4818275b","Type":"ContainerStarted","Data":"01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00"} Oct 30 02:05:57.027990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:57.027635 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" event={"ID":"e41e030d-cd33-4bca-9cdc-f72c4818275b","Type":"ContainerStarted","Data":"a4cfb33a37cb80c5e0ca9e04cd5aa1c565e43998241d8489eaceb7f8cefb1735"} Oct 30 02:05:57.093103 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:57.093047 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podStartSLOduration=2.093035897 podStartE2EDuration="2.093035897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:57.081140414 +0000 UTC m=+2185.472948041" watchObservedRunningTime="2026-10-30 02:05:57.093035897 +0000 UTC m=+2185.484843529" Oct 30 02:05:57.668115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:57.668082 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:57.673753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:57.673730 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:58.034210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:58.034176 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:05:58.035146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:05:58.035108 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:06:01.258632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.258597 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5"] Oct 30 02:06:01.262824 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.262805 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.276462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.276435 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-428529\"" Oct 30 02:06:01.286766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.286734 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-428529-dockercfg-8wng5\"" Oct 30 02:06:01.290307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.290277 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5"] Oct 30 02:06:01.313274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.313238 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txw6\" (UniqueName: \"kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6\") pod \"payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.313571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.313547 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.438150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.416001 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.452145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.438795 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txw6\" (UniqueName: \"kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6\") pod \"payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.452601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.439370 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.496144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.480613 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txw6\" (UniqueName: \"kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6\") pod \"payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.577430 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.577396 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:01.883963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:01.883929 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5"] Oct 30 02:06:01.886953 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:06:01.886928 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9f861f_e986_40e4_b678_dc25d80d98f6.slice/crio-a3ab7cc3538fd236e43757468053e78c143b7cde5e631da20a8dedf3e72f70cb WatchSource:0}: Error finding container a3ab7cc3538fd236e43757468053e78c143b7cde5e631da20a8dedf3e72f70cb: Status 404 returned error can't find the container with id a3ab7cc3538fd236e43757468053e78c143b7cde5e631da20a8dedf3e72f70cb Oct 30 02:06:02.062265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:02.062217 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" event={"ID":"cd9f861f-e986-40e4-b678-dc25d80d98f6","Type":"ContainerStarted","Data":"b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9"} Oct 30 02:06:02.062460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:02.062312 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" event={"ID":"cd9f861f-e986-40e4-b678-dc25d80d98f6","Type":"ContainerStarted","Data":"a3ab7cc3538fd236e43757468053e78c143b7cde5e631da20a8dedf3e72f70cb"} Oct 30 02:06:02.062460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:02.062338 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:02.109028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:02.108968 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" podStartSLOduration=1.10895031 podStartE2EDuration="1.10895031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:02.103928616 +0000 UTC m=+2190.495736249" watchObservedRunningTime="2026-10-30 02:06:02.10895031 +0000 UTC m=+2190.500757943" Oct 30 02:06:05.970148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:05.970090 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld"] Oct 30 02:06:05.970551 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:05.970330 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" podUID="dce93002-2643-4955-b305-667b7b5ee6ec" containerName="payload-processing" containerID="cri-o://e01b39e2a0a67e75f33dc5edb1e19375c05d5f17822592fb982970c823ed3466" gracePeriod=30 Oct 30 02:06:05.998615 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:05.998582 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc"] Oct 30 02:06:05.999022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:05.998970 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" podUID="4cff8538-4e13-426a-a33f-2fbb3d6c4397" containerName="payload-pre-processing" containerID="cri-o://337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42" gracePeriod=30 Oct 30 02:06:06.118259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.118222 6275 generic.go:356] "Generic (PLEG): container finished" podID="dce93002-2643-4955-b305-667b7b5ee6ec" containerID="e01b39e2a0a67e75f33dc5edb1e19375c05d5f17822592fb982970c823ed3466" exitCode=0 Oct 30 02:06:06.118259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.118249 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" event={"ID":"dce93002-2643-4955-b305-667b7b5ee6ec","Type":"ContainerDied","Data":"e01b39e2a0a67e75f33dc5edb1e19375c05d5f17822592fb982970c823ed3466"} Oct 30 02:06:06.304774 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.304741 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:06:06.468146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.466938 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:06:06.476156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.475786 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume\") pod \"dce93002-2643-4955-b305-667b7b5ee6ec\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " Oct 30 02:06:06.476156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.476017 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw\" (UniqueName: \"kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw\") pod \"dce93002-2643-4955-b305-667b7b5ee6ec\" (UID: \"dce93002-2643-4955-b305-667b7b5ee6ec\") " Oct 30 02:06:06.492511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.492390 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume" pod "dce93002-2643-4955-b305-667b7b5ee6ec" (UID: "dce93002-2643-4955-b305-667b7b5ee6ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:06.523624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.523579 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw" pod "dce93002-2643-4955-b305-667b7b5ee6ec" (UID: "dce93002-2643-4955-b305-667b7b5ee6ec"). InnerVolumeSpecName "kube-api-access-lq5rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:06.602259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.602221 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz\" (UniqueName: \"kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz\") pod \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " Oct 30 02:06:06.602796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.602360 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume\") pod \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\" (UID: \"4cff8538-4e13-426a-a33f-2fbb3d6c4397\") " Oct 30 02:06:06.602796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.602733 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume" pod "4cff8538-4e13-426a-a33f-2fbb3d6c4397" (UID: "4cff8538-4e13-426a-a33f-2fbb3d6c4397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:06.603094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.603064 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff8538-4e13-426a-a33f-2fbb3d6c4397-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.603094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.603086 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce93002-2643-4955-b305-667b7b5ee6ec-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.603234 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.603096 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq5rw\" (UniqueName: \"kubernetes.io/projected/dce93002-2643-4955-b305-667b7b5ee6ec-kube-api-access-lq5rw\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.604766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.604744 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz" pod "4cff8538-4e13-426a-a33f-2fbb3d6c4397" (UID: "4cff8538-4e13-426a-a33f-2fbb3d6c4397"). InnerVolumeSpecName "kube-api-access-7v8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:06.739147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:06.717615 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8xz\" (UniqueName: \"kubernetes.io/projected/4cff8538-4e13-426a-a33f-2fbb3d6c4397-kube-api-access-7v8xz\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:07.125913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.125881 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" Oct 30 02:06:07.125913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.125897 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld" event={"ID":"dce93002-2643-4955-b305-667b7b5ee6ec","Type":"ContainerDied","Data":"824390c094b5c41c27cf330e60ad40819dbce9e49ec818c511fff3437113ef3b"} Oct 30 02:06:07.126449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.125962 6275 scope.go:122] "RemoveContainer" containerID="e01b39e2a0a67e75f33dc5edb1e19375c05d5f17822592fb982970c823ed3466" Oct 30 02:06:07.127722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.127686 6275 generic.go:356] "Generic (PLEG): container finished" podID="4cff8538-4e13-426a-a33f-2fbb3d6c4397" containerID="337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42" exitCode=0 Oct 30 02:06:07.127844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.127752 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" event={"ID":"4cff8538-4e13-426a-a33f-2fbb3d6c4397","Type":"ContainerDied","Data":"337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42"} Oct 30 02:06:07.127844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.127773 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" Oct 30 02:06:07.127844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.127827 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc" event={"ID":"4cff8538-4e13-426a-a33f-2fbb3d6c4397","Type":"ContainerDied","Data":"5069f1b7c37bfdf710d983c1d6c72e8cc4c3dff58b9cbbf7bf9f9cc38e47326e"} Oct 30 02:06:07.136792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.136769 6275 scope.go:122] "RemoveContainer" containerID="337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42" Oct 30 02:06:07.145829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.145807 6275 scope.go:122] "RemoveContainer" containerID="337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42" Oct 30 02:06:07.146105 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:06:07.146080 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42\": container with ID starting with 337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42 not found: ID does not exist" containerID="337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42" Oct 30 02:06:07.146212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.146115 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42"} err="failed to get container status \"337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42\": rpc error: code = NotFound desc = could not find container \"337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42\": container with ID starting with 337cd13a2825cde0d28b1ee1e5c7bfbd91dc54923f49387816787ef8a37e4b42 not found: ID does not exist" Oct 30 02:06:07.191690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.191661 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc"] Oct 30 02:06:07.204165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.204045 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-54fd699959-xqjsc"] Oct 30 02:06:07.235489 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.235460 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld"] Oct 30 02:06:07.240813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.240741 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f3cd8-55989f95fb-rn9ld"] Oct 30 02:06:07.739220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.739186 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cff8538-4e13-426a-a33f-2fbb3d6c4397" path="/var/lib/kubelet/pods/4cff8538-4e13-426a-a33f-2fbb3d6c4397/volumes" Oct 30 02:06:07.739605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:07.739591 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce93002-2643-4955-b305-667b7b5ee6ec" path="/var/lib/kubelet/pods/dce93002-2643-4955-b305-667b7b5ee6ec/volumes" Oct 30 02:06:10.985290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:10.985253 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv"] Oct 30 02:06:10.986113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:10.986076 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" containerID="cri-o://b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78" gracePeriod=30 Oct 30 02:06:13.102355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.102327 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:06:13.459037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.458946 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl"] Oct 30 02:06:13.459570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.459545 6275 cpu_manager.go:414] "containerMap: removing container" podUID="dce93002-2643-4955-b305-667b7b5ee6ec" containerName="payload-processing" Oct 30 02:06:13.459570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.459566 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce93002-2643-4955-b305-667b7b5ee6ec" containerName="payload-processing" Oct 30 02:06:13.459767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.459587 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4cff8538-4e13-426a-a33f-2fbb3d6c4397" containerName="payload-pre-processing" Oct 30 02:06:13.459767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.459593 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cff8538-4e13-426a-a33f-2fbb3d6c4397" containerName="payload-pre-processing" Oct 30 02:06:13.459767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.459687 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce93002-2643-4955-b305-667b7b5ee6ec" containerName="payload-processing" Oct 30 02:06:13.459767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.459705 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cff8538-4e13-426a-a33f-2fbb3d6c4397" containerName="payload-pre-processing" Oct 30 02:06:13.464957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.464937 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.487254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.487219 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-428529-serving-cert\"" Oct 30 02:06:13.491523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.491333 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl"] Oct 30 02:06:13.569066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.569033 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.569256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.569080 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.569256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.569210 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6tz\" (UniqueName: \"kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.569359 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.569340 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.673630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.673586 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.674066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.674045 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.674302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.674280 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.674466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.674449 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6tz\" (UniqueName: \"kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.684917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.684889 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.714197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.710784 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.714197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.711703 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.723900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.723862 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6tz\" (UniqueName: \"kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz\") pod \"maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:13.780261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:13.780234 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:14.093527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:14.093399 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl"] Oct 30 02:06:14.096049 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:06:14.096023 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4569df95_06ee_44ae_bfc0_331eeb04976a.slice/crio-8433a27a3907110a1fe2098a1e7602cf340022bb2efadf0a3f3de74d687c5a8a WatchSource:0}: Error finding container 8433a27a3907110a1fe2098a1e7602cf340022bb2efadf0a3f3de74d687c5a8a: Status 404 returned error can't find the container with id 8433a27a3907110a1fe2098a1e7602cf340022bb2efadf0a3f3de74d687c5a8a Oct 30 02:06:14.097421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:14.097402 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:06:14.262019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:14.261984 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" event={"ID":"4569df95-06ee-44ae-bfc0-331eeb04976a","Type":"ContainerStarted","Data":"8433a27a3907110a1fe2098a1e7602cf340022bb2efadf0a3f3de74d687c5a8a"} Oct 30 02:06:15.271037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:15.270990 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" event={"ID":"4569df95-06ee-44ae-bfc0-331eeb04976a","Type":"ContainerStarted","Data":"d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4"} Oct 30 02:06:15.271473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:15.271349 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:15.339146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:15.329867 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" podStartSLOduration=1.8862300319999998 podStartE2EDuration="2.329840753s" totalImagesPullingTime="443.610721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:15.319558194 +0000 UTC m=+2203.711365828" watchObservedRunningTime="2026-10-30 02:06:15.329840753 +0000 UTC m=+2203.721648386" Oct 30 02:06:16.255850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.255824 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:06:16.278978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.278947 6275 generic.go:356] "Generic (PLEG): container finished" podID="18c3d3c4-718f-4eef-ae21-124902a580be" containerID="b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78" exitCode=0 Oct 30 02:06:16.279403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.278991 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" Oct 30 02:06:16.279403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.279019 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" event={"ID":"18c3d3c4-718f-4eef-ae21-124902a580be","Type":"ContainerDied","Data":"b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78"} Oct 30 02:06:16.279403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.279059 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv" event={"ID":"18c3d3c4-718f-4eef-ae21-124902a580be","Type":"ContainerDied","Data":"81bed210221ac52c5fc37e694935013f7e4600ec4e270ca40ae1d4aecce7e419"} Oct 30 02:06:16.279403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.279081 6275 scope.go:122] "RemoveContainer" containerID="b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78" Oct 30 02:06:16.288651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.288634 6275 scope.go:122] "RemoveContainer" containerID="666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0" Oct 30 02:06:16.297347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.297331 6275 scope.go:122] "RemoveContainer" containerID="b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78" Oct 30 02:06:16.297604 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:06:16.297585 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78\": container with ID starting with b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78 not found: ID does not exist" containerID="b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78" Oct 30 02:06:16.297657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.297613 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78"} err="failed to get container status \"b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78\": rpc error: code = NotFound desc = could not find container \"b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78\": container with ID starting with b9100b634e9d19ce4926c8367ad6d02a0c84930b62a6fa2047c26d242c80eb78 not found: ID does not exist" Oct 30 02:06:16.297657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.297629 6275 scope.go:122] "RemoveContainer" containerID="666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0" Oct 30 02:06:16.297852 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:06:16.297835 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0\": container with ID starting with 666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0 not found: ID does not exist" containerID="666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0" Oct 30 02:06:16.297900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.297859 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0"} err="failed to get container status \"666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0\": rpc error: code = NotFound desc = could not find container \"666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0\": container with ID starting with 666a1b52875556756d7e7a32e5729e9ecdbe3507da33cddb934bd0e1648642a0 not found: ID does not exist" Oct 30 02:06:16.327369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.327322 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.328973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.328540 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.329328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.329308 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.329427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.329368 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.329968 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.329698 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:16.329968 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.329712 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:16.329968 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.329930 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:16.330665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.330352 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.331140 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.330811 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.331355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.331332 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.331499 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.331367 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.331499 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.331448 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert\") pod \"18c3d3c4-718f-4eef-ae21-124902a580be\" (UID: \"18c3d3c4-718f-4eef-ae21-124902a580be\") " Oct 30 02:06:16.334428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.332173 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.334428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.332196 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.334428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.332208 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.334428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.332203 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "kube-api-access-mxg4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:16.334428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.332241 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:16.334428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.333650 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:16.335453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.335429 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:16.335453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.335436 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:16.335573 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.335535 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token" pod "18c3d3c4-718f-4eef-ae21-124902a580be" (UID: "18c3d3c4-718f-4eef-ae21-124902a580be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:16.433640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.433498 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c3d3c4-718f-4eef-ae21-124902a580be-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.433640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.433532 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxg4f\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-kube-api-access-mxg4f\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.433640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.433549 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c3d3c4-718f-4eef-ae21-124902a580be-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.433640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.433564 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.433640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.433578 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c3d3c4-718f-4eef-ae21-124902a580be-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.433640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.433592 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c3d3c4-718f-4eef-ae21-124902a580be-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.662275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.662232 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv"] Oct 30 02:06:16.665528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:16.665492 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f3cd8-openshift-default-55c4fc49b5-vdjbv"] Oct 30 02:06:16.777112 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:06:16.777037 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:06:17.781601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:17.781558 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" path="/var/lib/kubelet/pods/18c3d3c4-718f-4eef-ae21-124902a580be/volumes" Oct 30 02:06:18.637248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:18.637203 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:18.732266 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:18.732212 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:18.750265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:18.750217 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:21.285302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:21.285270 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:06:31.088804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.088774 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp"] Oct 30 02:06:31.089558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.089531 6275 cpu_manager.go:414] "containerMap: removing container" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" Oct 30 02:06:31.089558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.089552 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" Oct 30 02:06:31.089730 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.089677 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" Oct 30 02:06:31.093248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.093220 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.120005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.119975 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp"] Oct 30 02:06:31.167740 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.167703 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.167896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.167751 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.167896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.167771 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vp4\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.167978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.167921 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.167978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.167968 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.168043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.168025 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.168077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.168052 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.168139 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.168098 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.168232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.168214 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.268980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.268946 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.268994 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269021 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vp4\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269106 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269176 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269202 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269233 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269304 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269388 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269549 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269730 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269770 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.269935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.269911 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.270251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.270232 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.271444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.271416 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.271797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.271779 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.290988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.290964 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.291105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.291058 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vp4\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.407446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.407367 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:31.740845 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:06:31.740764 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4bf0e9_1776_49b5_b76c_aa51d4b0d4d6.slice/crio-5893e8ac99230a09ed4b6b4694d79ea695706b7bc8234f6e55bbfdc0489a282a WatchSource:0}: Error finding container 5893e8ac99230a09ed4b6b4694d79ea695706b7bc8234f6e55bbfdc0489a282a: Status 404 returned error can't find the container with id 5893e8ac99230a09ed4b6b4694d79ea695706b7bc8234f6e55bbfdc0489a282a Oct 30 02:06:31.741754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.741732 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp"] Oct 30 02:06:31.743549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.743519 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:31.743640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.743585 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:31.743640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:31.743611 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:32.464087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:32.464048 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" event={"ID":"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6","Type":"ContainerStarted","Data":"67d38c7c442bc0cd42e457be00aec4ade7849e3950c8a0f971661fcb15dcfb66"} Oct 30 02:06:32.464505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:32.464162 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" event={"ID":"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6","Type":"ContainerStarted","Data":"5893e8ac99230a09ed4b6b4694d79ea695706b7bc8234f6e55bbfdc0489a282a"} Oct 30 02:06:32.528788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:32.528737 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podStartSLOduration=1.5287234010000001 podStartE2EDuration="1.528723401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:32.516995309 +0000 UTC m=+2220.908802943" watchObservedRunningTime="2026-10-30 02:06:32.528723401 +0000 UTC m=+2220.920531108" Oct 30 02:06:33.407767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:33.407726 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:33.409590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:33.409565 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:33.409733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:33.409613 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:34.410150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:34.409790 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:34.410150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:34.409855 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:34.415724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:34.413941 6275 scope.go:122] "RemoveContainer" containerID="c260cd67eadd5caf8dd565f19ba26e407d59145cfc16078c0963085f9960b6fd" Oct 30 02:06:34.458391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:34.452385 6275 scope.go:122] "RemoveContainer" containerID="d1616d4cc25717a1d18c6931e399f54dee2eceea9c62b9163cd39a9eb0798e0e" Oct 30 02:06:35.408319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:35.408285 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:35.408506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:35.408344 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:36.408031 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:36.407994 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:36.408566 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:36.408073 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:37.408347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:37.408315 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:37.408809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:37.408368 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:38.420171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:38.410568 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:38.420171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:38.410631 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:39.424144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:39.423736 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:06:39.424144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:39.423809 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 02:06:40.610488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:40.609243 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:40.643390 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:40.643251 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:40.739146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:40.735388 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 02:06:40.760579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:40.760533 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:40.790302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:40.790253 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:41.459957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.459312 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:41.459957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.459385 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:41.471821 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.471608 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:41.516751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.516683 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:41.519051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.518963 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 02:06:41.519376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.517579 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:41.615488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.613208 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 02:06:41.615488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.613954 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 02:06:41.650146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.649247 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 02:06:41.650146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.650000 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 02:06:41.753379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.753216 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:41.754395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.754360 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 02:06:41.843147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.818402 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": context deadline exceeded" Oct 30 02:06:41.908571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.849791 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 02:06:41.934523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.901810 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:41.966148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:41.943213 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:42.232207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.232180 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:06:42.232407 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.232306 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 02:06:42.232494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.232388 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:42.232494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.232481 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 02:06:42.235219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.235186 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 02:06:42.235363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.235243 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:42.237563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.237545 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:06:42.290524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.290303 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:06:42.294290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.291753 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 02:06:42.297173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:42.297010 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 02:06:43.576149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.575380 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz"] Oct 30 02:06:43.576663 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.576390 6275 cpu_manager.go:414] "containerMap: removing container" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" Oct 30 02:06:43.576663 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.576413 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" Oct 30 02:06:43.576663 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.576606 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c3d3c4-718f-4eef-ae21-124902a580be" containerName="istio-proxy" Oct 30 02:06:43.582698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.582669 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.592778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.592744 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz"] Oct 30 02:06:43.596419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.596396 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b4a561\"" Oct 30 02:06:43.596661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.596636 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b4a561-dockercfg-gxqfb\"" Oct 30 02:06:43.621838 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.621809 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm"] Oct 30 02:06:43.627721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.627698 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:43.676484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.676455 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm"] Oct 30 02:06:43.701549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.701518 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdbz\" (UniqueName: \"kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz\") pod \"payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.701711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.701641 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.817148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.812990 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.817148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.813268 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwmc\" (UniqueName: \"kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc\") pod \"payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:43.817148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.813340 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:43.817148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.813451 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdbz\" (UniqueName: \"kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz\") pod \"payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.817148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.814671 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.954195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.921833 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdbz\" (UniqueName: \"kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz\") pod \"payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:43.954195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.922589 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwmc\" (UniqueName: \"kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc\") pod \"payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:43.954195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.931533 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:43.954195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:43.945422 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:44.009628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:44.009562 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwmc\" (UniqueName: \"kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc\") pod \"payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:44.205322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:44.205219 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:44.288652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:44.288628 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:44.578383 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:44.578351 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz"] Oct 30 02:06:44.631714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:44.631515 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" event={"ID":"a6fbc7d9-955c-47fd-87f8-895a5450dd35","Type":"ContainerStarted","Data":"d020691b68a41baca14449ff185ae0a145de9eeec8345d9a020b18f81212908b"} Oct 30 02:06:44.633505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:44.633484 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm"] Oct 30 02:06:44.635969 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:06:44.635942 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35231403_960c_4191_ab71_6849909dd10a.slice/crio-868626b4254e6ea8b352c23f360b7002f0185bce4ccdc7cc195717d135008cda WatchSource:0}: Error finding container 868626b4254e6ea8b352c23f360b7002f0185bce4ccdc7cc195717d135008cda: Status 404 returned error can't find the container with id 868626b4254e6ea8b352c23f360b7002f0185bce4ccdc7cc195717d135008cda Oct 30 02:06:45.665584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:45.665533 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" event={"ID":"35231403-960c-4191-ab71-6849909dd10a","Type":"ContainerStarted","Data":"15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53"} Oct 30 02:06:45.666069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:45.665607 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" event={"ID":"35231403-960c-4191-ab71-6849909dd10a","Type":"ContainerStarted","Data":"868626b4254e6ea8b352c23f360b7002f0185bce4ccdc7cc195717d135008cda"} Oct 30 02:06:45.666069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:45.665641 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:45.667487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:45.667459 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" event={"ID":"a6fbc7d9-955c-47fd-87f8-895a5450dd35","Type":"ContainerStarted","Data":"5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164"} Oct 30 02:06:45.667620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:45.667593 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:45.705376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:45.705317 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" podStartSLOduration=2.705300715 podStartE2EDuration="2.705300715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:45.702974321 +0000 UTC m=+2234.094781951" watchObservedRunningTime="2026-10-30 02:06:45.705300715 +0000 UTC m=+2234.097108374" Oct 30 02:06:46.706919 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:06:46.706831 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:06:49.932680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.932631 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" podStartSLOduration=6.932616342 podStartE2EDuration="6.932616342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:45.76470521 +0000 UTC m=+2234.156512844" watchObservedRunningTime="2026-10-30 02:06:49.932616342 +0000 UTC m=+2238.324424021" Oct 30 02:06:49.934474 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.934449 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs"] Oct 30 02:06:49.938792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.938770 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.965069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.965043 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs"] Oct 30 02:06:49.990550 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.990513 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.990723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.990643 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.990790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.990747 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjl4x\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.990927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.990900 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.990984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.990952 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.991061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.991045 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.991161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.991145 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.991252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.991229 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:49.991308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:49.991293 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.092927 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjl4x\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.092986 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093010 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093041 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093085 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093158 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093209 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093282 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093352 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093470 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093503 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093582 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.093942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.093914 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.094168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.094134 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.095719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.095698 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.096091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.096070 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.114703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.114647 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.114877 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.114858 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjl4x\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.257484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.257457 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:50.583635 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.583608 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs"] Oct 30 02:06:50.586229 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:06:50.586196 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304167dd_cf30_452f_bfce_b5c7bb64ed10.slice/crio-cfb6115a54d946db1c19e1dcd340576099edfc000e48844f38b9deba06efce2c WatchSource:0}: Error finding container cfb6115a54d946db1c19e1dcd340576099edfc000e48844f38b9deba06efce2c: Status 404 returned error can't find the container with id cfb6115a54d946db1c19e1dcd340576099edfc000e48844f38b9deba06efce2c Oct 30 02:06:50.588836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.588801 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:50.588926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.588870 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:50.588926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.588898 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:50.706734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.706700 6275 generic.go:356] "Generic (PLEG): container finished" podID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerID="01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00" exitCode=0 Oct 30 02:06:50.706932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.706778 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" event={"ID":"e41e030d-cd33-4bca-9cdc-f72c4818275b","Type":"ContainerDied","Data":"01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00"} Oct 30 02:06:50.707295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.707274 6275 scope.go:122] "RemoveContainer" containerID="01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00" Oct 30 02:06:50.708786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.708763 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" event={"ID":"304167dd-cf30-452f-bfce-b5c7bb64ed10","Type":"ContainerStarted","Data":"774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c"} Oct 30 02:06:50.708904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.708803 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" event={"ID":"304167dd-cf30-452f-bfce-b5c7bb64ed10","Type":"ContainerStarted","Data":"cfb6115a54d946db1c19e1dcd340576099edfc000e48844f38b9deba06efce2c"} Oct 30 02:06:50.709057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.709022 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:50.709227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.709111 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:50.709227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.709184 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:50.798227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:50.798179 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podStartSLOduration=1.798164341 podStartE2EDuration="1.798164341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:50.792634022 +0000 UTC m=+2239.184441684" watchObservedRunningTime="2026-10-30 02:06:50.798164341 +0000 UTC m=+2239.189971974" Oct 30 02:06:51.260458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.260425 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:06:51.264178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.264148 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:51.264318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.264201 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:51.401087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.401048 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn"] Oct 30 02:06:51.406578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.406555 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.422224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.422185 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-4153e5-openshift-default-dockercfg-5jm4n\"" Oct 30 02:06:51.429089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.429069 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn"] Oct 30 02:06:51.521807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.521713 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.521807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.521762 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.521882 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.521945 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bjc\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.522018 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.522155 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.522198 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.522246 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.522379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.522342 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625053 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bjc\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625151 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625267 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625313 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625378 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625487 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625588 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625627 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.625691 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.626229 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.626832 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.627871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.627852 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.628790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.628111 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.628790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.628617 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.649147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.642901 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.677148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.675785 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.681142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.677530 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.693209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.688843 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bjc\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc\") pod \"e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.767144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.760311 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:51.867147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:51.865139 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" event={"ID":"e41e030d-cd33-4bca-9cdc-f72c4818275b","Type":"ContainerStarted","Data":"010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61"} Oct 30 02:06:52.291196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.290585 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:52.291196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.290654 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:52.295931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.295902 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn"] Oct 30 02:06:52.297617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.297587 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq"] Oct 30 02:06:52.299492 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:06:52.299465 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d1c41c_7055_41c5_9a20_3a1b41dd709b.slice/crio-7eb6353a25e4f52de0dbf9a068364f808c70b26a4ff2e09500edea5839bd963f WatchSource:0}: Error finding container 7eb6353a25e4f52de0dbf9a068364f808c70b26a4ff2e09500edea5839bd963f: Status 404 returned error can't find the container with id 7eb6353a25e4f52de0dbf9a068364f808c70b26a4ff2e09500edea5839bd963f Oct 30 02:06:52.304412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.304310 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:52.305955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.305866 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:52.305955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.305930 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:06:52.673138 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.672588 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:06:52.673138 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.672760 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:52.673138 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.672807 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:52.860837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.860800 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" event={"ID":"68d1c41c-7055-41c5-9a20-3a1b41dd709b","Type":"ContainerStarted","Data":"3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e"} Oct 30 02:06:52.890428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.890386 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" event={"ID":"68d1c41c-7055-41c5-9a20-3a1b41dd709b","Type":"ContainerStarted","Data":"7eb6353a25e4f52de0dbf9a068364f808c70b26a4ff2e09500edea5839bd963f"} Oct 30 02:06:52.913235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:52.913099 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podStartSLOduration=1.913080238 podStartE2EDuration="1.913080238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:52.911672508 +0000 UTC m=+2241.303480142" watchObservedRunningTime="2026-10-30 02:06:52.913080238 +0000 UTC m=+2241.304887872" Oct 30 02:06:53.264599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.264564 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:53.264866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.264838 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:53.675199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.670503 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:53.675199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.670563 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:53.776188 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.770611 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:53.794146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.790409 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:53.882177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.880240 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:53.882177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:53.881387 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:06:54.266151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:54.263477 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:54.266151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:54.263543 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:54.692203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:54.691221 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:54.692203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:54.691311 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:55.238149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.237272 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:55.242244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.239956 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:55.273793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.273749 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:55.274065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.274036 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:55.292584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.292530 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:55.306151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.303232 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:55.669950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.669914 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:55.670271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.670240 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:55.692973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.692921 6275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 02:06:55.747985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:55.747943 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:56.241239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.241190 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": context deadline exceeded" Oct 30 02:06:56.241806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.241216 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:56.242645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.242621 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:06:56.264145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.242825 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 02:06:56.264671 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.263262 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:56.264916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.264886 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:56.311300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.311254 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:56.311942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.311904 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 02:06:56.312426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.312406 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 02:06:56.313521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.313503 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 02:06:56.626147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.609230 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:56.626147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.609320 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:56.688198 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.688163 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:56.688494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.688449 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:56.770316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.749444 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:56.770316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.750229 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 02:06:56.775358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.775333 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:06:56.787651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.787452 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:06:56.878648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.878552 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:56.878932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:56.878900 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:57.258207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.244329 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:57.258207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.244688 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:57.282886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.277150 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:57.282886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.277207 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:57.316147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.312157 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:57.340146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.317470 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 02:06:57.677279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.677244 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:57.677600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.677553 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:57.738606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.724556 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 02:06:57.738606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.737375 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:06:57.738606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.738102 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 02:06:57.824023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.823991 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 02:06:57.824593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:57.824576 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 02:06:58.034214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.034181 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:06:58.279384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.279351 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:58.279669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.279642 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:58.673292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.673256 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 02:06:58.673901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.673870 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 02:06:58.774176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.758456 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522"] Oct 30 02:06:58.790151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:58.774626 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" containerID="cri-o://64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410" gracePeriod=30 Oct 30 02:06:59.258732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.258693 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:06:59.259183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.258767 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:06:59.597239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.595859 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg"] Oct 30 02:06:59.601229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.600739 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.625382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.625343 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-2819d2-openshift-default-dockercfg-6vq2z\"" Oct 30 02:06:59.646862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.646820 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg"] Oct 30 02:06:59.680077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.680030 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:06:59.700214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.699477 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:06:59.802159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.794102 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvx4\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.802467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.802436 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.802638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.802619 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.802847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.802826 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.803082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.803064 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.803283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.803264 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.803439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.803424 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.803600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.803584 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.810150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.803744 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.919044 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.918947 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919475 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919557 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919608 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919662 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919740 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvx4\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919793 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919817 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.933912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.919909 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.980171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.966037 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.980171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.966622 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.980171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.967290 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.980171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.969192 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:06:59.980171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:06:59.977449 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:00.024346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.024290 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:00.034994 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.034961 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:00.036997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.036876 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvx4\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:00.036997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.036924 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy\") pod \"e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:00.217859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.217763 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:00.258274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.258242 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:07:00.258423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.258294 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 02:07:00.554720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.554688 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg"] Oct 30 02:07:00.555971 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:00.555940 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf789454e_6ef6_47d1_b92d_b18248f1ab9b.slice/crio-467987cde086a732183814c264e29baef84bb3c4b033a9dc668d4e6738d22fb4 WatchSource:0}: Error finding container 467987cde086a732183814c264e29baef84bb3c4b033a9dc668d4e6738d22fb4: Status 404 returned error can't find the container with id 467987cde086a732183814c264e29baef84bb3c4b033a9dc668d4e6738d22fb4 Oct 30 02:07:00.559274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.559234 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:00.559358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.559322 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:00.559399 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:00.559370 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:01.431703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:01.424358 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" event={"ID":"f789454e-6ef6-47d1-b92d-b18248f1ab9b","Type":"ContainerStarted","Data":"0b0ae3f2a23417584746ac29b577d2786519fa523b6a3bf578e135c65b545a88"} Oct 30 02:07:01.431703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:01.424428 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" event={"ID":"f789454e-6ef6-47d1-b92d-b18248f1ab9b","Type":"ContainerStarted","Data":"467987cde086a732183814c264e29baef84bb3c4b033a9dc668d4e6738d22fb4"} Oct 30 02:07:01.493939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:01.493866 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podStartSLOduration=2.493846027 podStartE2EDuration="2.493846027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:01.481694216 +0000 UTC m=+2249.873501858" watchObservedRunningTime="2026-10-30 02:07:01.493846027 +0000 UTC m=+2249.885653661" Oct 30 02:07:02.218809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.218774 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:02.224374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.224348 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:02.303144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.302251 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:07:02.303144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.302322 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:02.433031 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.432994 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:02.433773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.433752 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:07:02.451189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.451162 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:07:02.451454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.451432 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:07:02.452305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:02.452285 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:07:04.094246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.094184 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 02:07:04.118473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118446 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118538 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118602 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118687 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118733 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118784 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118818 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.118866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118836 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:04.118866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118851 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.119075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.118917 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket\") pod \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\" (UID: \"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4\") " Oct 30 02:07:04.119150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119083 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:04.119307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119277 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:04.119385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119303 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:04.119385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119317 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:04.119577 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119552 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.119695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119579 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.119695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119592 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.119695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119604 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.119695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.119615 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.120934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.120911 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "kube-api-access-bm2pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:04.121013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.120948 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:04.121209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.121187 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:07:04.121295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.121222 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy" pod "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" (UID: "9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:04.221200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.221169 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.221200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.221194 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.221200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.221204 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.221435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.221213 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2pt\" (UniqueName: \"kubernetes.io/projected/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4-kube-api-access-bm2pt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:04.448870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.448784 6275 generic.go:356] "Generic (PLEG): container finished" podID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerID="64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410" exitCode=0 Oct 30 02:07:04.448870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.448843 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" Oct 30 02:07:04.449226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.448865 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" event={"ID":"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4","Type":"ContainerDied","Data":"64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410"} Oct 30 02:07:04.449226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.448924 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522" event={"ID":"9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4","Type":"ContainerDied","Data":"50144469352ce75793dfdd57e3835733d96e3b2b68a4ec15b0be635153258e29"} Oct 30 02:07:04.449226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.448950 6275 scope.go:122] "RemoveContainer" containerID="64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410" Oct 30 02:07:04.460045 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.460019 6275 scope.go:122] "RemoveContainer" containerID="64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410" Oct 30 02:07:04.460385 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:07:04.460360 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410\": container with ID starting with 64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410 not found: ID does not exist" containerID="64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410" Oct 30 02:07:04.460457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.460398 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410"} err="failed to get container status \"64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410\": rpc error: code = NotFound desc = could not find container \"64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410\": container with ID starting with 64caf38f62eef8c49f2292b37773f09eb48429fff968fc69058746a32d854410 not found: ID does not exist" Oct 30 02:07:04.499476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.499451 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522"] Oct 30 02:07:04.509701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.509679 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nw522"] Oct 30 02:07:04.918606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.918569 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48"] Oct 30 02:07:04.919448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.919422 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" Oct 30 02:07:04.919448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.919445 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" Oct 30 02:07:04.919655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.919606 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" containerName="istio-proxy" Oct 30 02:07:04.924646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.924624 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:04.947563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.947541 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2979be-openshift-default-dockercfg-jmq5l\"" Oct 30 02:07:04.951378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:04.951325 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48"] Oct 30 02:07:05.031568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.031530 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tws\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035291 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035356 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035445 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035506 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035553 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035649 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035700 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.035789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.035731 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.136851 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.136817 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tws\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.136887 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.136922 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.136938 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.136967 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137170 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137294 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137354 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137391 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137404 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137452 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137493 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.137685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.137682 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.138018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.138001 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.139515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.139484 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.139629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.139531 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.159014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.158988 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tws\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.164913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.164889 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.239069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.238983 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:05.625726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.625692 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48"] Oct 30 02:07:05.628173 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:05.628143 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffe3f1b_44fa_4df7_af4f_86acfc1fe5d4.slice/crio-3293aa0a3b743e708bad9af680ff9b2749c8bd96639e6de474765c676c9b9819 WatchSource:0}: Error finding container 3293aa0a3b743e708bad9af680ff9b2749c8bd96639e6de474765c676c9b9819: Status 404 returned error can't find the container with id 3293aa0a3b743e708bad9af680ff9b2749c8bd96639e6de474765c676c9b9819 Oct 30 02:07:05.630909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.630879 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:05.630988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.630944 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:05.630988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.630975 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:05.741833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.741798 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4" path="/var/lib/kubelet/pods/9484f878-ba1a-4ae6-a3c4-20aaf1d3cdf4/volumes" Oct 30 02:07:05.767921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.767882 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc"] Oct 30 02:07:05.772463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.772441 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:05.791494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.791468 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-2819d2-dockercfg-fq6jt\"" Oct 30 02:07:05.795844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.795798 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc"] Oct 30 02:07:05.827046 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.827003 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-2819d2\"" Oct 30 02:07:05.904681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.902972 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:05.904681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.903231 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmll\" (UniqueName: \"kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll\") pod \"payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:05.961315 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.961276 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm"] Oct 30 02:07:05.986329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.979513 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm"] Oct 30 02:07:05.986329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:05.979633 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.057629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.057589 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:06.057944 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.057919 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.058167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.058147 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfpt\" (UniqueName: \"kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt\") pod \"payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.058352 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.058328 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmll\" (UniqueName: \"kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll\") pod \"payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:06.059912 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.059878 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:06.094189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.094138 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmll\" (UniqueName: \"kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll\") pod \"payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:06.159911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.159720 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.159911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.159874 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfpt\" (UniqueName: \"kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt\") pod \"payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.160813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.160788 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.181938 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.181911 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfpt\" (UniqueName: \"kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt\") pod \"payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.377206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.377175 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:06.390493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.390467 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:06.489297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.489156 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" event={"ID":"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4","Type":"ContainerStarted","Data":"3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6"} Oct 30 02:07:06.489297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.489271 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" event={"ID":"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4","Type":"ContainerStarted","Data":"3293aa0a3b743e708bad9af680ff9b2749c8bd96639e6de474765c676c9b9819"} Oct 30 02:07:06.532248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.532197 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podStartSLOduration=2.532181718 podStartE2EDuration="2.532181718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:06.530999705 +0000 UTC m=+2254.922807341" watchObservedRunningTime="2026-10-30 02:07:06.532181718 +0000 UTC m=+2254.923989351" Oct 30 02:07:06.695728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.695704 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm"] Oct 30 02:07:06.697447 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:06.697420 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd3212b_a086_47bf_802f_ced6540a3d9b.slice/crio-156c0a37c602a194d098820cd6073553631eef853152cc952c66c0d6b6437d9b WatchSource:0}: Error finding container 156c0a37c602a194d098820cd6073553631eef853152cc952c66c0d6b6437d9b: Status 404 returned error can't find the container with id 156c0a37c602a194d098820cd6073553631eef853152cc952c66c0d6b6437d9b Oct 30 02:07:06.761147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:06.739718 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc"] Oct 30 02:07:06.773932 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:06.773895 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9ff630_5b88_44db_af58_15ca56ed2389.slice/crio-b9df171a73175aba5dd0e30ad5c3ed1acd7716be00fa3e83ad86e32f7bdd9526 WatchSource:0}: Error finding container b9df171a73175aba5dd0e30ad5c3ed1acd7716be00fa3e83ad86e32f7bdd9526: Status 404 returned error can't find the container with id b9df171a73175aba5dd0e30ad5c3ed1acd7716be00fa3e83ad86e32f7bdd9526 Oct 30 02:07:07.242147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.239229 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:07.254148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.244427 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 02:07:07.254148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.244485 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 02:07:07.327679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.327620 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" containerID="cri-o://2ce5335b324d54b2d08bf65a5c73e2de287af99162fc7f70cc41f0da3efc07a6" gracePeriod=45 Oct 30 02:07:07.500505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.500421 6275 generic.go:356] "Generic (PLEG): container finished" podID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerID="2ce5335b324d54b2d08bf65a5c73e2de287af99162fc7f70cc41f0da3efc07a6" exitCode=0 Oct 30 02:07:07.500630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.500502 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" event={"ID":"ed97dca7-1fc7-413b-89a0-d683114bc41d","Type":"ContainerDied","Data":"2ce5335b324d54b2d08bf65a5c73e2de287af99162fc7f70cc41f0da3efc07a6"} Oct 30 02:07:07.502312 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.502284 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" event={"ID":"5f9ff630-5b88-44db-af58-15ca56ed2389","Type":"ContainerStarted","Data":"139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88"} Oct 30 02:07:07.502455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.502336 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" event={"ID":"5f9ff630-5b88-44db-af58-15ca56ed2389","Type":"ContainerStarted","Data":"b9df171a73175aba5dd0e30ad5c3ed1acd7716be00fa3e83ad86e32f7bdd9526"} Oct 30 02:07:07.502455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.502404 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:07.504097 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.504071 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" event={"ID":"8cd3212b-a086-47bf-802f-ced6540a3d9b","Type":"ContainerStarted","Data":"82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201"} Oct 30 02:07:07.504234 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.504116 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" event={"ID":"8cd3212b-a086-47bf-802f-ced6540a3d9b","Type":"ContainerStarted","Data":"156c0a37c602a194d098820cd6073553631eef853152cc952c66c0d6b6437d9b"} Oct 30 02:07:07.570223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.569992 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" podStartSLOduration=2.569975188 podStartE2EDuration="2.569975188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:07.565603009 +0000 UTC m=+2255.957410645" watchObservedRunningTime="2026-10-30 02:07:07.569975188 +0000 UTC m=+2255.961782865" Oct 30 02:07:07.605418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.605359 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" podStartSLOduration=2.6053415920000003 podStartE2EDuration="2.605341592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:07.603310725 +0000 UTC m=+2255.995118360" watchObservedRunningTime="2026-10-30 02:07:07.605341592 +0000 UTC m=+2255.997149226" Oct 30 02:07:07.689235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.689205 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:07:07.699273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699246 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs\" (UniqueName: \"kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs\") pod \"ed97dca7-1fc7-413b-89a0-d683114bc41d\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " Oct 30 02:07:07.699400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699290 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home\") pod \"ed97dca7-1fc7-413b-89a0-d683114bc41d\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " Oct 30 02:07:07.699400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699333 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache\") pod \"ed97dca7-1fc7-413b-89a0-d683114bc41d\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " Oct 30 02:07:07.699400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699368 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir\") pod \"ed97dca7-1fc7-413b-89a0-d683114bc41d\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " Oct 30 02:07:07.699540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699450 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs\") pod \"ed97dca7-1fc7-413b-89a0-d683114bc41d\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " Oct 30 02:07:07.699604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699578 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm\") pod \"ed97dca7-1fc7-413b-89a0-d683114bc41d\" (UID: \"ed97dca7-1fc7-413b-89a0-d683114bc41d\") " Oct 30 02:07:07.699664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699592 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache" pod "ed97dca7-1fc7-413b-89a0-d683114bc41d" (UID: "ed97dca7-1fc7-413b-89a0-d683114bc41d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:07.699664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699604 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home" pod "ed97dca7-1fc7-413b-89a0-d683114bc41d" (UID: "ed97dca7-1fc7-413b-89a0-d683114bc41d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:07.699768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.699696 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir" pod "ed97dca7-1fc7-413b-89a0-d683114bc41d" (UID: "ed97dca7-1fc7-413b-89a0-d683114bc41d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:07.700164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.700117 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:07.700164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.700165 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:07.700375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.700178 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:07.701690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.701659 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs" pod "ed97dca7-1fc7-413b-89a0-d683114bc41d" (UID: "ed97dca7-1fc7-413b-89a0-d683114bc41d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:07.701794 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.701668 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm" pod "ed97dca7-1fc7-413b-89a0-d683114bc41d" (UID: "ed97dca7-1fc7-413b-89a0-d683114bc41d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:07.701794 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.701709 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs" pod "ed97dca7-1fc7-413b-89a0-d683114bc41d" (UID: "ed97dca7-1fc7-413b-89a0-d683114bc41d"). InnerVolumeSpecName "kube-api-access-457qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:07.812816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.812780 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed97dca7-1fc7-413b-89a0-d683114bc41d-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:07.818228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.818199 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-457qs\" (UniqueName: \"kubernetes.io/projected/ed97dca7-1fc7-413b-89a0-d683114bc41d-kube-api-access-457qs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:07.818228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:07.818229 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed97dca7-1fc7-413b-89a0-d683114bc41d-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:08.267148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.264581 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 02:07:08.267148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.264651 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 02:07:08.513307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.513279 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" Oct 30 02:07:08.513498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.513309 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq" event={"ID":"ed97dca7-1fc7-413b-89a0-d683114bc41d","Type":"ContainerDied","Data":"4d29f52c62d7696656b27d2716a5e574a9473523581acf004471d9b190c6ef98"} Oct 30 02:07:08.513498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.513365 6275 scope.go:122] "RemoveContainer" containerID="2ce5335b324d54b2d08bf65a5c73e2de287af99162fc7f70cc41f0da3efc07a6" Oct 30 02:07:08.513617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.513574 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:08.590942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.590906 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq"] Oct 30 02:07:08.596932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:08.596884 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-2979be/praxis-route-2979be-kserve-57c6798677-8mnvq"] Oct 30 02:07:09.255282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:09.255243 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 02:07:09.255579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:09.255549 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 02:07:09.771918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:09.771870 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" path="/var/lib/kubelet/pods/ed97dca7-1fc7-413b-89a0-d683114bc41d/volumes" Oct 30 02:07:10.245207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:10.243488 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 02:07:10.245207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:10.243555 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 02:07:11.243154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:11.242478 6275 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 02:07:11.243154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:11.242547 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 02:07:12.259651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:12.259569 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:12.260586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:12.260561 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:12.262541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:12.262520 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:07:15.433726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.433690 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8"] Oct 30 02:07:15.434360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.434336 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" Oct 30 02:07:15.434420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.434361 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" Oct 30 02:07:15.434485 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.434476 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed97dca7-1fc7-413b-89a0-d683114bc41d" containerName="main" Oct 30 02:07:15.439393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.439373 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.457475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.457452 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-4153e5-serving-cert\"" Oct 30 02:07:15.468040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.468009 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8"] Oct 30 02:07:15.595758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.595724 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.595923 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.595793 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.595923 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.595884 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4j2g\" (UniqueName: \"kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.596044 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.596024 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.698306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.698204 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.699310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.699280 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.699933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.699899 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.707287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.707255 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4j2g\" (UniqueName: \"kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.708138 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.700825 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.726456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.726420 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.740342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.740307 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4j2g\" (UniqueName: \"kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.748223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.748189 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:15.755033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:15.754991 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:16.106827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:16.106796 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8"] Oct 30 02:07:16.109153 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:16.108646 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9b6424_2631_48fb_a12c_42dd5303a752.slice/crio-0bcfa960248a43a3a5d04fd985a265c6c1e718eee5188002ef0e9c623cbf536b WatchSource:0}: Error finding container 0bcfa960248a43a3a5d04fd985a265c6c1e718eee5188002ef0e9c623cbf536b: Status 404 returned error can't find the container with id 0bcfa960248a43a3a5d04fd985a265c6c1e718eee5188002ef0e9c623cbf536b Oct 30 02:07:16.630979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:16.630943 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" event={"ID":"ae9b6424-2631-48fb-a12c-42dd5303a752","Type":"ContainerStarted","Data":"0bcfa960248a43a3a5d04fd985a265c6c1e718eee5188002ef0e9c623cbf536b"} Oct 30 02:07:16.699165 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:07:16.699101 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:07:17.638763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:17.638726 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" event={"ID":"ae9b6424-2631-48fb-a12c-42dd5303a752","Type":"ContainerStarted","Data":"4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c"} Oct 30 02:07:17.639194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:17.638793 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:17.686645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:17.686595 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" podStartSLOduration=2.16350111 podStartE2EDuration="2.686583297s" totalImagesPullingTime="523.082187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:17.684332326 +0000 UTC m=+2266.076139958" watchObservedRunningTime="2026-10-30 02:07:17.686583297 +0000 UTC m=+2266.078390929" Oct 30 02:07:18.517618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:18.517585 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:07:19.573145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:19.560383 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:07:23.087909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.084289 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m"] Oct 30 02:07:23.091694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.091669 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.121562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.121230 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-2819d2-serving-cert\"" Oct 30 02:07:23.130596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.130082 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m"] Oct 30 02:07:23.229733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.229695 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.229908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.229812 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.229969 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.229907 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9vk\" (UniqueName: \"kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.230053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.230039 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.341207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.334079 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.341207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.334263 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9vk\" (UniqueName: \"kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.341207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.334402 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.341207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.334503 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.341207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.336055 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.381142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.379903 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.381142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.379991 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.402034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.400536 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9vk\" (UniqueName: \"kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk\") pod \"maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.412547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.412492 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:23.660807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.660734 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:07:23.722092 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:23.722061 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc564e47b_d931_4e9f_a58e_00d3f9d314a9.slice/crio-d4abc5b05b3fc914c3c304f50d831eb4bba924e6dca48dba7f14b53067de34e6 WatchSource:0}: Error finding container d4abc5b05b3fc914c3c304f50d831eb4bba924e6dca48dba7f14b53067de34e6: Status 404 returned error can't find the container with id d4abc5b05b3fc914c3c304f50d831eb4bba924e6dca48dba7f14b53067de34e6 Oct 30 02:07:23.725566 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.725536 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m"] Oct 30 02:07:23.831881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:23.831845 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" event={"ID":"c564e47b-d931-4e9f-a58e-00d3f9d314a9","Type":"ContainerStarted","Data":"d4abc5b05b3fc914c3c304f50d831eb4bba924e6dca48dba7f14b53067de34e6"} Oct 30 02:07:24.839865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:24.839824 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" event={"ID":"c564e47b-d931-4e9f-a58e-00d3f9d314a9","Type":"ContainerStarted","Data":"19f5399a46a7d5663e2578648ec76d2c278de684200fa41ea2703349dd679125"} Oct 30 02:07:24.840377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:24.839913 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:24.881700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:24.881643 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" podStartSLOduration=1.523338534 podStartE2EDuration="1.881603733s" totalImagesPullingTime="358.265199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:24.87875055 +0000 UTC m=+2273.270558184" watchObservedRunningTime="2026-10-30 02:07:24.881603733 +0000 UTC m=+2273.273411366" Oct 30 02:07:30.862480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:30.862447 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:07:31.171084 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:31.170999 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v"] Oct 30 02:07:35.139241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:35.139212 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5"] Oct 30 02:07:36.171521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:36.171481 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerName="payload-pre-processing" containerID="cri-o://4218302e0ec8b00278f66d67e5f85096a84a4b2f789912acd092d2975d3183f0" gracePeriod=25 Oct 30 02:07:37.402996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:37.402957 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:07:40.140218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.140171 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" podUID="cd9f861f-e986-40e4-b678-dc25d80d98f6" containerName="payload-pre-processing" containerID="cri-o://b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9" gracePeriod=25 Oct 30 02:07:40.427918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.427888 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:07:40.526035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.526002 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume\") pod \"cd9f861f-e986-40e4-b678-dc25d80d98f6\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " Oct 30 02:07:40.526242 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.526077 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6\" (UniqueName: \"kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6\") pod \"cd9f861f-e986-40e4-b678-dc25d80d98f6\" (UID: \"cd9f861f-e986-40e4-b678-dc25d80d98f6\") " Oct 30 02:07:40.526487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.526455 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume" pod "cd9f861f-e986-40e4-b678-dc25d80d98f6" (UID: "cd9f861f-e986-40e4-b678-dc25d80d98f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:40.526654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.526636 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9f861f-e986-40e4-b678-dc25d80d98f6-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:40.528255 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.528222 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6" pod "cd9f861f-e986-40e4-b678-dc25d80d98f6" (UID: "cd9f861f-e986-40e4-b678-dc25d80d98f6"). InnerVolumeSpecName "kube-api-access-7txw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:40.628506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:40.628472 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txw6\" (UniqueName: \"kubernetes.io/projected/cd9f861f-e986-40e4-b678-dc25d80d98f6-kube-api-access-7txw6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:41.114738 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.114703 6275 generic.go:356] "Generic (PLEG): container finished" podID="cd9f861f-e986-40e4-b678-dc25d80d98f6" containerID="b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9" exitCode=0 Oct 30 02:07:41.114941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.114804 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" event={"ID":"cd9f861f-e986-40e4-b678-dc25d80d98f6","Type":"ContainerDied","Data":"b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9"} Oct 30 02:07:41.114941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.114855 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" event={"ID":"cd9f861f-e986-40e4-b678-dc25d80d98f6","Type":"ContainerDied","Data":"a3ab7cc3538fd236e43757468053e78c143b7cde5e631da20a8dedf3e72f70cb"} Oct 30 02:07:41.114941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.114882 6275 scope.go:122] "RemoveContainer" containerID="b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9" Oct 30 02:07:41.115116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.115065 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5" Oct 30 02:07:41.143613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.143332 6275 scope.go:122] "RemoveContainer" containerID="b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9" Oct 30 02:07:41.143976 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:07:41.143764 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9\": container with ID starting with b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9 not found: ID does not exist" containerID="b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9" Oct 30 02:07:41.143976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.143800 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9"} err="failed to get container status \"b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9\": rpc error: code = NotFound desc = could not find container \"b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9\": container with ID starting with b9b3b031765857d33207248962531aa4ef456c28fd632ffcc0acb86b30aa63f9 not found: ID does not exist" Oct 30 02:07:41.171224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.171189 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5"] Oct 30 02:07:41.175266 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.175240 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-428529-8679b59df9-4nfl5"] Oct 30 02:07:41.751574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:41.751539 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9f861f-e986-40e4-b678-dc25d80d98f6" path="/var/lib/kubelet/pods/cd9f861f-e986-40e4-b678-dc25d80d98f6/volumes" Oct 30 02:07:46.699534 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:07:46.699497 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:07:47.402669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:47.402628 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:07:50.534994 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.534956 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8"] Oct 30 02:07:50.536043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.536022 6275 cpu_manager.go:414] "containerMap: removing container" podUID="cd9f861f-e986-40e4-b678-dc25d80d98f6" containerName="payload-pre-processing" Oct 30 02:07:50.536104 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.536044 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9f861f-e986-40e4-b678-dc25d80d98f6" containerName="payload-pre-processing" Oct 30 02:07:50.536189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.536180 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9f861f-e986-40e4-b678-dc25d80d98f6" containerName="payload-pre-processing" Oct 30 02:07:50.563147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.563034 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:50.569248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.569215 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8"] Oct 30 02:07:50.578940 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.578917 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-17e747\"" Oct 30 02:07:50.579105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.578996 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-17e747-dockercfg-zfzwm\"" Oct 30 02:07:50.632233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.632201 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:50.632407 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.632383 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzjx\" (UniqueName: \"kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx\") pod \"payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:50.698146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.697626 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk"] Oct 30 02:07:50.714493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.714458 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:50.742149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.722210 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk"] Oct 30 02:07:50.743533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.743491 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:50.743918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.743894 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzjx\" (UniqueName: \"kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx\") pod \"payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:50.744189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.744162 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:50.744407 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.744388 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbhk\" (UniqueName: \"kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk\") pod \"payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:50.745775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.745750 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:50.909217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.899185 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:50.909217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.899316 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbhk\" (UniqueName: \"kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk\") pod \"payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:50.932262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.932214 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:51.000079 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:50.996303 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzjx\" (UniqueName: \"kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx\") pod \"payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:51.013727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:51.013684 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbhk\" (UniqueName: \"kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk\") pod \"payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:51.184044 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:51.183954 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:51.304935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:51.304897 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:51.503339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:51.502838 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8"] Oct 30 02:07:51.504271 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:51.504241 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b79155_0055_4bfa_9fc3_cdb24a0344d7.slice/crio-59f9e2ec904ba442487c27b6c1851bf3b7188ef3375fd0972c273d3b1ea15382 WatchSource:0}: Error finding container 59f9e2ec904ba442487c27b6c1851bf3b7188ef3375fd0972c273d3b1ea15382: Status 404 returned error can't find the container with id 59f9e2ec904ba442487c27b6c1851bf3b7188ef3375fd0972c273d3b1ea15382 Oct 30 02:07:51.669223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:51.669167 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk"] Oct 30 02:07:51.671272 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:51.671244 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01d5808_adaf_4887_96f1_6319afa759b7.slice/crio-cc257c306cc3d38bee8cb7ab5e0fa7375f32f730b2e0ba9045ba442d1d96309e WatchSource:0}: Error finding container cc257c306cc3d38bee8cb7ab5e0fa7375f32f730b2e0ba9045ba442d1d96309e: Status 404 returned error can't find the container with id cc257c306cc3d38bee8cb7ab5e0fa7375f32f730b2e0ba9045ba442d1d96309e Oct 30 02:07:52.282335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.282296 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" event={"ID":"b01d5808-adaf-4887-96f1-6319afa759b7","Type":"ContainerStarted","Data":"511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95"} Oct 30 02:07:52.282531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.282452 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:07:52.282531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.282476 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" event={"ID":"b01d5808-adaf-4887-96f1-6319afa759b7","Type":"ContainerStarted","Data":"cc257c306cc3d38bee8cb7ab5e0fa7375f32f730b2e0ba9045ba442d1d96309e"} Oct 30 02:07:52.283758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.283732 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" event={"ID":"70b79155-0055-4bfa-9fc3-cdb24a0344d7","Type":"ContainerStarted","Data":"b4e80e8094d6176c60f578a61aad1904047b8d440c0f135fd641913720a013d2"} Oct 30 02:07:52.283894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.283773 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" event={"ID":"70b79155-0055-4bfa-9fc3-cdb24a0344d7","Type":"ContainerStarted","Data":"59f9e2ec904ba442487c27b6c1851bf3b7188ef3375fd0972c273d3b1ea15382"} Oct 30 02:07:52.283894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.283792 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:07:52.325524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.325467 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" podStartSLOduration=2.325452019 podStartE2EDuration="2.325452019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:52.322297916 +0000 UTC m=+2300.714105550" watchObservedRunningTime="2026-10-30 02:07:52.325452019 +0000 UTC m=+2300.717259657" Oct 30 02:07:52.364677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:52.364625 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" podStartSLOduration=2.364609668 podStartE2EDuration="2.364609668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:52.363674598 +0000 UTC m=+2300.755482233" watchObservedRunningTime="2026-10-30 02:07:52.364609668 +0000 UTC m=+2300.756417320" Oct 30 02:07:53.535142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:53.534974 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl"] Oct 30 02:07:53.577488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:53.577442 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" podUID="4569df95-06ee-44ae-bfc0-331eeb04976a" containerName="maas-api" containerID="cri-o://d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4" gracePeriod=30 Oct 30 02:07:54.211054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.211027 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:07:54.379159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.376879 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca\") pod \"4569df95-06ee-44ae-bfc0-331eeb04976a\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " Oct 30 02:07:54.379159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.376970 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls\") pod \"4569df95-06ee-44ae-bfc0-331eeb04976a\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " Oct 30 02:07:54.379159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.377099 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs\") pod \"4569df95-06ee-44ae-bfc0-331eeb04976a\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " Oct 30 02:07:54.379159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.377207 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz\" (UniqueName: \"kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz\") pod \"4569df95-06ee-44ae-bfc0-331eeb04976a\" (UID: \"4569df95-06ee-44ae-bfc0-331eeb04976a\") " Oct 30 02:07:54.379555 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.379454 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca" pod "4569df95-06ee-44ae-bfc0-331eeb04976a" (UID: "4569df95-06ee-44ae-bfc0-331eeb04976a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:54.396042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.395097 6275 generic.go:356] "Generic (PLEG): container finished" podID="4569df95-06ee-44ae-bfc0-331eeb04976a" containerID="d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4" exitCode=0 Oct 30 02:07:54.396042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.395371 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" event={"ID":"4569df95-06ee-44ae-bfc0-331eeb04976a","Type":"ContainerDied","Data":"d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4"} Oct 30 02:07:54.396042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.395429 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" event={"ID":"4569df95-06ee-44ae-bfc0-331eeb04976a","Type":"ContainerDied","Data":"8433a27a3907110a1fe2098a1e7602cf340022bb2efadf0a3f3de74d687c5a8a"} Oct 30 02:07:54.396042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.395459 6275 scope.go:122] "RemoveContainer" containerID="d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4" Oct 30 02:07:54.396042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.395648 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl" Oct 30 02:07:54.479725 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.479488 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569df95-06ee-44ae-bfc0-331eeb04976a-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:54.482777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.482744 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs" pod "4569df95-06ee-44ae-bfc0-331eeb04976a" (UID: "4569df95-06ee-44ae-bfc0-331eeb04976a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:54.483668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.483626 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls" pod "4569df95-06ee-44ae-bfc0-331eeb04976a" (UID: "4569df95-06ee-44ae-bfc0-331eeb04976a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:54.485941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.485624 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz" pod "4569df95-06ee-44ae-bfc0-331eeb04976a" (UID: "4569df95-06ee-44ae-bfc0-331eeb04976a"). InnerVolumeSpecName "kube-api-access-nh6tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:54.500738 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.500707 6275 scope.go:122] "RemoveContainer" containerID="d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4" Oct 30 02:07:54.501164 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:07:54.501111 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4\": container with ID starting with d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4 not found: ID does not exist" containerID="d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4" Oct 30 02:07:54.501286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.501165 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4"} err="failed to get container status \"d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4\": rpc error: code = NotFound desc = could not find container \"d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4\": container with ID starting with d468d16d733d7c0d8b1e42869413aeeff6aafa9fd43dab9bfe594c2d0f8e6cf4 not found: ID does not exist" Oct 30 02:07:54.581745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.581708 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:54.582117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.581758 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4569df95-06ee-44ae-bfc0-331eeb04976a-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:54.582117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.581775 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6tz\" (UniqueName: \"kubernetes.io/projected/4569df95-06ee-44ae-bfc0-331eeb04976a-kube-api-access-nh6tz\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:54.738909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.738877 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl"] Oct 30 02:07:54.746074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:54.746047 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-428529-5d5b494547-cmlnl"] Oct 30 02:07:55.740897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:55.740858 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4569df95-06ee-44ae-bfc0-331eeb04976a" path="/var/lib/kubelet/pods/4569df95-06ee-44ae-bfc0-331eeb04976a/volumes" Oct 30 02:07:56.416001 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.415960 6275 generic.go:356] "Generic (PLEG): container finished" podID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerID="4218302e0ec8b00278f66d67e5f85096a84a4b2f789912acd092d2975d3183f0" exitCode=0 Oct 30 02:07:56.416253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.416049 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" event={"ID":"2fc4203c-bc41-4b06-9175-f43ccf6e7a94","Type":"ContainerDied","Data":"4218302e0ec8b00278f66d67e5f85096a84a4b2f789912acd092d2975d3183f0"} Oct 30 02:07:56.477520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.477489 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:07:56.613295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.613257 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume\") pod \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " Oct 30 02:07:56.613494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.613336 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk\" (UniqueName: \"kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk\") pod \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\" (UID: \"2fc4203c-bc41-4b06-9175-f43ccf6e7a94\") " Oct 30 02:07:56.613656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.613630 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume" pod "2fc4203c-bc41-4b06-9175-f43ccf6e7a94" (UID: "2fc4203c-bc41-4b06-9175-f43ccf6e7a94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:56.613990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.613971 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:56.615544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.615519 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk" pod "2fc4203c-bc41-4b06-9175-f43ccf6e7a94" (UID: "2fc4203c-bc41-4b06-9175-f43ccf6e7a94"). InnerVolumeSpecName "kube-api-access-nmmfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:56.716281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:56.716202 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmmfk\" (UniqueName: \"kubernetes.io/projected/2fc4203c-bc41-4b06-9175-f43ccf6e7a94-kube-api-access-nmmfk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:07:57.192144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.191615 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr"] Oct 30 02:07:57.192641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.192610 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerName="payload-pre-processing" Oct 30 02:07:57.192641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.192632 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerName="payload-pre-processing" Oct 30 02:07:57.192763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.192659 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4569df95-06ee-44ae-bfc0-331eeb04976a" containerName="maas-api" Oct 30 02:07:57.192763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.192666 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4569df95-06ee-44ae-bfc0-331eeb04976a" containerName="maas-api" Oct 30 02:07:57.212144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.192858 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" containerName="payload-pre-processing" Oct 30 02:07:57.212144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.192881 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4569df95-06ee-44ae-bfc0-331eeb04976a" containerName="maas-api" Oct 30 02:07:57.212144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.197105 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.228780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.227773 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9a1cd0\"" Oct 30 02:07:57.271746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.271711 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9a1cd0-dockercfg-jwq2m\"" Oct 30 02:07:57.283973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.283856 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr"] Oct 30 02:07:57.334366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.334325 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.334546 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.334507 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dqb\" (UniqueName: \"kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb\") pod \"payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.426653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.426622 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" Oct 30 02:07:57.426653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.426645 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v" event={"ID":"2fc4203c-bc41-4b06-9175-f43ccf6e7a94","Type":"ContainerDied","Data":"d2f580de500467b096211e546c21f79d8a440c2b133c38cf08a20c6c93dffdd9"} Oct 30 02:07:57.426874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.426706 6275 scope.go:122] "RemoveContainer" containerID="4218302e0ec8b00278f66d67e5f85096a84a4b2f789912acd092d2975d3183f0" Oct 30 02:07:57.435577 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.435537 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.435715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.435702 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dqb\" (UniqueName: \"kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb\") pod \"payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.436240 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.436222 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.457475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.457421 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dqb\" (UniqueName: \"kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb\") pod \"payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.474826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.474796 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v"] Oct 30 02:07:57.478450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.478425 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2979be-d6d4dd6cf-jls4v"] Oct 30 02:07:57.578091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.578040 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:57.741352 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.741277 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc4203c-bc41-4b06-9175-f43ccf6e7a94" path="/var/lib/kubelet/pods/2fc4203c-bc41-4b06-9175-f43ccf6e7a94/volumes" Oct 30 02:07:57.893701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:57.893674 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr"] Oct 30 02:07:57.896521 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:07:57.896490 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1828b46c_1215_4b27_84a8_32d2e7f3f56b.slice/crio-74124afa233fbc03b4f9ce90bfb2c1ac09c467aca6684876aa29b70a67c69220 WatchSource:0}: Error finding container 74124afa233fbc03b4f9ce90bfb2c1ac09c467aca6684876aa29b70a67c69220: Status 404 returned error can't find the container with id 74124afa233fbc03b4f9ce90bfb2c1ac09c467aca6684876aa29b70a67c69220 Oct 30 02:07:58.436619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.436527 6275 generic.go:356] "Generic (PLEG): container finished" podID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerID="0b0ae3f2a23417584746ac29b577d2786519fa523b6a3bf578e135c65b545a88" exitCode=0 Oct 30 02:07:58.436619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.436599 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" event={"ID":"f789454e-6ef6-47d1-b92d-b18248f1ab9b","Type":"ContainerDied","Data":"0b0ae3f2a23417584746ac29b577d2786519fa523b6a3bf578e135c65b545a88"} Oct 30 02:07:58.437167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.437113 6275 scope.go:122] "RemoveContainer" containerID="0b0ae3f2a23417584746ac29b577d2786519fa523b6a3bf578e135c65b545a88" Oct 30 02:07:58.438323 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.438296 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" event={"ID":"1828b46c-1215-4b27-84a8-32d2e7f3f56b","Type":"ContainerStarted","Data":"7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f"} Oct 30 02:07:58.438439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.438346 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" event={"ID":"1828b46c-1215-4b27-84a8-32d2e7f3f56b","Type":"ContainerStarted","Data":"74124afa233fbc03b4f9ce90bfb2c1ac09c467aca6684876aa29b70a67c69220"} Oct 30 02:07:58.438439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.438377 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:07:58.438904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.438872 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:58.438980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.438962 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:58.439041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.439014 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:07:58.528545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:58.528496 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" podStartSLOduration=1.528483314 podStartE2EDuration="1.528483314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:58.52834392 +0000 UTC m=+2306.920151569" watchObservedRunningTime="2026-10-30 02:07:58.528483314 +0000 UTC m=+2306.920290946" Oct 30 02:07:59.471418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:07:59.471380 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" event={"ID":"f789454e-6ef6-47d1-b92d-b18248f1ab9b","Type":"ContainerStarted","Data":"6392b21109e286456f3f615a49c9c47bacba116912560c5c8bcea81ea9940e43"} Oct 30 02:08:00.218366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:00.218330 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:08:00.219294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:00.219270 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:08:00.219363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:00.219320 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 02:08:01.218273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:01.218234 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:08:01.218679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:01.218299 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 02:08:02.221809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:02.221773 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:08:02.222507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:02.222473 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 02:08:02.260529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:02.260369 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6"] Oct 30 02:08:02.273988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:02.273316 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" containerID="cri-o://010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61" gracePeriod=30 Oct 30 02:08:02.450163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:02.436223 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:08:03.218603 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:03.218571 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:08:03.218777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:03.218633 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 02:08:03.293533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:03.293506 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:08:03.294005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:03.293572 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:08:04.227061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:04.227020 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:08:04.227507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:04.227470 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 02:08:04.312726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:04.312694 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48"] Oct 30 02:08:04.313293 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:04.312898 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" containerID="cri-o://3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6" gracePeriod=30 Oct 30 02:08:05.218915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:05.218870 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:08:05.219181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:05.218927 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 02:08:06.222883 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:06.222854 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:08:06.223976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:06.223960 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:08:07.592146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.592070 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:08:07.614678 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.614650 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.614846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.614707 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.614846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.614771 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.614846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.614820 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.615037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.614927 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.615037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.614952 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.615037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615025 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.615253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615099 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.615253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615177 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy\") pod \"e41e030d-cd33-4bca-9cdc-f72c4818275b\" (UID: \"e41e030d-cd33-4bca-9cdc-f72c4818275b\") " Oct 30 02:08:07.615253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615213 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:07.615497 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615473 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:07.616055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615765 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:07.616055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615974 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.616055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.615995 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.616055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.616009 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.616432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.616403 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:07.616600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.616579 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:07.617506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.617464 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:08:07.617816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.617796 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "kube-api-access-wn7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:07.617891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.617825 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:07.617891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.617833 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy" pod "e41e030d-cd33-4bca-9cdc-f72c4818275b" (UID: "e41e030d-cd33-4bca-9cdc-f72c4818275b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:07.717650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.717617 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.717650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.717650 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7ll\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-kube-api-access-wn7ll\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.717870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.717666 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.717870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.717677 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.717870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.717690 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e030d-cd33-4bca-9cdc-f72c4818275b-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.717870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.717701 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e030d-cd33-4bca-9cdc-f72c4818275b-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:07.719111 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.719084 6275 generic.go:356] "Generic (PLEG): container finished" podID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerID="010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61" exitCode=0 Oct 30 02:08:07.719245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.719157 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" Oct 30 02:08:07.719245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.719172 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" event={"ID":"e41e030d-cd33-4bca-9cdc-f72c4818275b","Type":"ContainerDied","Data":"010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61"} Oct 30 02:08:07.719245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.719233 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6" event={"ID":"e41e030d-cd33-4bca-9cdc-f72c4818275b","Type":"ContainerDied","Data":"a4cfb33a37cb80c5e0ca9e04cd5aa1c565e43998241d8489eaceb7f8cefb1735"} Oct 30 02:08:07.719396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.719256 6275 scope.go:122] "RemoveContainer" containerID="010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61" Oct 30 02:08:07.730549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.730531 6275 scope.go:122] "RemoveContainer" containerID="01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00" Oct 30 02:08:07.740927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.740901 6275 scope.go:122] "RemoveContainer" containerID="010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61" Oct 30 02:08:07.741367 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:07.741345 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61\": container with ID starting with 010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61 not found: ID does not exist" containerID="010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61" Oct 30 02:08:07.741447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.741373 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61"} err="failed to get container status \"010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61\": rpc error: code = NotFound desc = could not find container \"010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61\": container with ID starting with 010f14d90c9ee834c4714874e0c6664350626a6378723ac0fff478152ac3de61 not found: ID does not exist" Oct 30 02:08:07.741447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.741390 6275 scope.go:122] "RemoveContainer" containerID="01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00" Oct 30 02:08:07.741650 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:07.741632 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00\": container with ID starting with 01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00 not found: ID does not exist" containerID="01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00" Oct 30 02:08:07.741705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.741661 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00"} err="failed to get container status \"01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00\": rpc error: code = NotFound desc = could not find container \"01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00\": container with ID starting with 01c236f554e94fc0c99869e27b4968762444ca303563b2bfc9c38b241f05db00 not found: ID does not exist" Oct 30 02:08:07.774222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.774190 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6"] Oct 30 02:08:07.777276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:07.777255 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-428529-openshift-default-5d4449bf97-9rwk6"] Oct 30 02:08:09.475685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.475658 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:08:09.596892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.596825 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:08:09.662021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.661977 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662044 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662099 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662149 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662198 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662481 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662332 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.662481 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662339 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662532 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.662596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662583 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662623 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662701 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs\") pod \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\" (UID: \"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4\") " Oct 30 02:08:09.662715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.662696 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.663028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.663001 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:09.663531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.663506 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.663634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.663537 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.663634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.663553 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.663634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.663568 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.663799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.663739 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.664929 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.664902 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "kube-api-access-z5tws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:09.664929 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.664904 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.665808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.665770 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:08:09.666071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.666043 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token" pod "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" (UID: "effe3f1b-44fa-4df7-af4f-86acfc1fe5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:09.779504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.779442 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.779504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.779476 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.779504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.779491 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.779504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.779504 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.779889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.779519 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tws\" (UniqueName: \"kubernetes.io/projected/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4-kube-api-access-z5tws\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.787923 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.787889 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" path="/var/lib/kubelet/pods/e41e030d-cd33-4bca-9cdc-f72c4818275b/volumes" Oct 30 02:08:09.802505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.802468 6275 generic.go:356] "Generic (PLEG): container finished" podID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerID="3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6" exitCode=0 Oct 30 02:08:09.802654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.802532 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" Oct 30 02:08:09.802654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.802557 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" event={"ID":"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4","Type":"ContainerDied","Data":"3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6"} Oct 30 02:08:09.802654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.802610 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48" event={"ID":"effe3f1b-44fa-4df7-af4f-86acfc1fe5d4","Type":"ContainerDied","Data":"3293aa0a3b743e708bad9af680ff9b2749c8bd96639e6de474765c676c9b9819"} Oct 30 02:08:09.802654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.802634 6275 scope.go:122] "RemoveContainer" containerID="3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6" Oct 30 02:08:09.813908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.813887 6275 scope.go:122] "RemoveContainer" containerID="3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6" Oct 30 02:08:09.814254 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:09.814223 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6\": container with ID starting with 3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6 not found: ID does not exist" containerID="3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6" Oct 30 02:08:09.814354 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.814267 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6"} err="failed to get container status \"3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6\": rpc error: code = NotFound desc = could not find container \"3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6\": container with ID starting with 3131f94776c5afb7476f9c982ee6c0c6fbfe6f73b6edb8072ab272e6dfa24ca6 not found: ID does not exist" Oct 30 02:08:09.853193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.853086 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48"] Oct 30 02:08:09.869592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:09.869564 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-fbz48"] Oct 30 02:08:10.422809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:10.422779 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm"] Oct 30 02:08:10.458160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:10.458113 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz"] Oct 30 02:08:11.741861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:11.741822 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" path="/var/lib/kubelet/pods/effe3f1b-44fa-4df7-af4f-86acfc1fe5d4/volumes" Oct 30 02:08:15.423453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.423412 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" podUID="35231403-960c-4191-ab71-6849909dd10a" containerName="payload-processing" containerID="cri-o://15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53" gracePeriod=25 Oct 30 02:08:15.458703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.458650 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" podUID="a6fbc7d9-955c-47fd-87f8-895a5450dd35" containerName="payload-pre-processing" containerID="cri-o://5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164" gracePeriod=25 Oct 30 02:08:15.745040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.745013 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:08:15.785795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.785765 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:08:15.811631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.811599 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume\") pod \"35231403-960c-4191-ab71-6849909dd10a\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " Oct 30 02:08:15.811795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.811734 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume\") pod \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " Oct 30 02:08:15.811850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.811840 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc\" (UniqueName: \"kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc\") pod \"35231403-960c-4191-ab71-6849909dd10a\" (UID: \"35231403-960c-4191-ab71-6849909dd10a\") " Oct 30 02:08:15.811979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.811962 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz\" (UniqueName: \"kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz\") pod \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\" (UID: \"a6fbc7d9-955c-47fd-87f8-895a5450dd35\") " Oct 30 02:08:15.812021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.812000 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume" pod "35231403-960c-4191-ab71-6849909dd10a" (UID: "35231403-960c-4191-ab71-6849909dd10a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:15.812427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.812403 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume" pod "a6fbc7d9-955c-47fd-87f8-895a5450dd35" (UID: "a6fbc7d9-955c-47fd-87f8-895a5450dd35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:15.812876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.812853 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35231403-960c-4191-ab71-6849909dd10a-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:15.812958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.812882 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fbc7d9-955c-47fd-87f8-895a5450dd35-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:15.815449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.815423 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc" pod "35231403-960c-4191-ab71-6849909dd10a" (UID: "35231403-960c-4191-ab71-6849909dd10a"). InnerVolumeSpecName "kube-api-access-crwmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:15.815599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.815582 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz" pod "a6fbc7d9-955c-47fd-87f8-895a5450dd35" (UID: "a6fbc7d9-955c-47fd-87f8-895a5450dd35"). InnerVolumeSpecName "kube-api-access-6pdbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:15.846248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.846220 6275 generic.go:356] "Generic (PLEG): container finished" podID="a6fbc7d9-955c-47fd-87f8-895a5450dd35" containerID="5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164" exitCode=0 Oct 30 02:08:15.846404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.846259 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" Oct 30 02:08:15.846404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.846305 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" event={"ID":"a6fbc7d9-955c-47fd-87f8-895a5450dd35","Type":"ContainerDied","Data":"5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164"} Oct 30 02:08:15.846404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.846358 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz" event={"ID":"a6fbc7d9-955c-47fd-87f8-895a5450dd35","Type":"ContainerDied","Data":"d020691b68a41baca14449ff185ae0a145de9eeec8345d9a020b18f81212908b"} Oct 30 02:08:15.846404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.846386 6275 scope.go:122] "RemoveContainer" containerID="5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164" Oct 30 02:08:15.848751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.848726 6275 generic.go:356] "Generic (PLEG): container finished" podID="35231403-960c-4191-ab71-6849909dd10a" containerID="15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53" exitCode=0 Oct 30 02:08:15.848921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.848753 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" Oct 30 02:08:15.848921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.848822 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" event={"ID":"35231403-960c-4191-ab71-6849909dd10a","Type":"ContainerDied","Data":"15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53"} Oct 30 02:08:15.848921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.848856 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm" event={"ID":"35231403-960c-4191-ab71-6849909dd10a","Type":"ContainerDied","Data":"868626b4254e6ea8b352c23f360b7002f0185bce4ccdc7cc195717d135008cda"} Oct 30 02:08:15.861266 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.861248 6275 scope.go:122] "RemoveContainer" containerID="5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164" Oct 30 02:08:15.862050 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:15.861980 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164\": container with ID starting with 5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164 not found: ID does not exist" containerID="5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164" Oct 30 02:08:15.862050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.862009 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164"} err="failed to get container status \"5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164\": rpc error: code = NotFound desc = could not find container \"5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164\": container with ID starting with 5bc8cb19eaeb4b9a2406256349e23b81e2e83dd51106b986c4c21564511a7164 not found: ID does not exist" Oct 30 02:08:15.862050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.862030 6275 scope.go:122] "RemoveContainer" containerID="15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53" Oct 30 02:08:15.871732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.871715 6275 scope.go:122] "RemoveContainer" containerID="15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53" Oct 30 02:08:15.871997 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:15.871978 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53\": container with ID starting with 15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53 not found: ID does not exist" containerID="15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53" Oct 30 02:08:15.872063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.872008 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53"} err="failed to get container status \"15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53\": rpc error: code = NotFound desc = could not find container \"15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53\": container with ID starting with 15c761c4baeafb41b2d3ddc8a24e2dcdcbac85aad83f3cf3c086d1a9a15f4f53 not found: ID does not exist" Oct 30 02:08:15.906867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.906832 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz"] Oct 30 02:08:15.910957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.910927 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b4a561-89c7c457f-h5vlz"] Oct 30 02:08:15.914748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.914723 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwmc\" (UniqueName: \"kubernetes.io/projected/35231403-960c-4191-ab71-6849909dd10a-kube-api-access-crwmc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:15.914748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.914748 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdbz\" (UniqueName: \"kubernetes.io/projected/a6fbc7d9-955c-47fd-87f8-895a5450dd35-kube-api-access-6pdbz\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:08:15.940711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.940682 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm"] Oct 30 02:08:15.944393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:15.944327 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b4a561-5c6655468-48jfm"] Oct 30 02:08:16.481308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.481274 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh"] Oct 30 02:08:16.481945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.481925 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" Oct 30 02:08:16.482029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.481947 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" Oct 30 02:08:16.482029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.481968 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a6fbc7d9-955c-47fd-87f8-895a5450dd35" containerName="payload-pre-processing" Oct 30 02:08:16.482029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.481975 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fbc7d9-955c-47fd-87f8-895a5450dd35" containerName="payload-pre-processing" Oct 30 02:08:16.482029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482005 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" Oct 30 02:08:16.482029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482013 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" Oct 30 02:08:16.482206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482034 6275 cpu_manager.go:414] "containerMap: removing container" podUID="35231403-960c-4191-ab71-6849909dd10a" containerName="payload-processing" Oct 30 02:08:16.482206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482039 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35231403-960c-4191-ab71-6849909dd10a" containerName="payload-processing" Oct 30 02:08:16.482206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482055 6275 cpu_manager.go:414] "containerMap: removing container" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" Oct 30 02:08:16.482206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482059 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" Oct 30 02:08:16.482206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482199 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" Oct 30 02:08:16.482363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482209 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fbc7d9-955c-47fd-87f8-895a5450dd35" containerName="payload-pre-processing" Oct 30 02:08:16.482363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482219 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="35231403-960c-4191-ab71-6849909dd10a" containerName="payload-processing" Oct 30 02:08:16.482363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.482226 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="effe3f1b-44fa-4df7-af4f-86acfc1fe5d4" containerName="istio-proxy" Oct 30 02:08:16.486552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.486530 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.504074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.504022 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9a1cd0-serving-cert\"" Oct 30 02:08:16.514585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.514558 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh"] Oct 30 02:08:16.523383 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.523351 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7rj2\" (UniqueName: \"kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.523574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.523527 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.523779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.523758 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.523880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.523860 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.625656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.625617 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7rj2\" (UniqueName: \"kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.625843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.625732 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.625939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.625922 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.626012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.625988 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.626113 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:16.626092 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-serving-cert: secret "maas-api-e2e-shared-b-w0-9a1cd0-serving-cert" not found Oct 30 02:08:16.626222 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:16.626207 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls podName:34ef869d-3809-420e-9622-3fb670d7cf8b nodeName:}" failed. No retries permitted until 2026-10-30 02:08:17.126186936 +0000 UTC m=+2325.517994561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls") pod "maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" (UID: "34ef869d-3809-420e-9622-3fb670d7cf8b") : secret "maas-api-e2e-shared-b-w0-9a1cd0-serving-cert" not found Oct 30 02:08:16.626436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.626415 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.628560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.628534 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.656333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:16.656300 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7rj2\" (UniqueName: \"kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:16.698831 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:16.698748 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:08:17.133091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.133059 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:17.135512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.135492 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:17.401340 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.401242 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:17.491539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.491502 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn"] Oct 30 02:08:17.492312 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.492291 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41e030d-cd33-4bca-9cdc-f72c4818275b" containerName="istio-proxy" Oct 30 02:08:17.496726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.496706 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.538364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.538335 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-594253-openshift-default-dockercfg-r8hqf\"" Oct 30 02:08:17.544694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.544661 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.544832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.544718 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc47r\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.544832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.544817 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.544928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.544909 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.545008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.544995 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.545056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.545023 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.545091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.545066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.545205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.545181 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.545262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.545251 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.565697 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.565672 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn"] Oct 30 02:08:17.684205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.671080 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.684414 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.684592 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.684680 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.684723 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc47r\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.684764 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.684891 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.685013 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.685053 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.711146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.685604 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.713904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.711839 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.728971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.714782 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.728971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.715516 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.728971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.727436 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.786144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.759960 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.787018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.770627 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.853697 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.853661 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35231403-960c-4191-ab71-6849909dd10a" path="/var/lib/kubelet/pods/35231403-960c-4191-ab71-6849909dd10a/volumes" Oct 30 02:08:17.854573 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.854552 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fbc7d9-955c-47fd-87f8-895a5450dd35" path="/var/lib/kubelet/pods/a6fbc7d9-955c-47fd-87f8-895a5450dd35/volumes" Oct 30 02:08:17.866527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.866485 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.877470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.877147 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc47r\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:17.904308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:17.901245 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh"] Oct 30 02:08:18.131149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.129000 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:18.473193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.473170 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn"] Oct 30 02:08:18.475380 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:08:18.475346 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc49b69_9b46_40d2_94aa_9c76bd29f433.slice/crio-ad449cf2dd30942e213fbc895fc95bf29f36c3d22c4a139be8c02d3da6a2a855 WatchSource:0}: Error finding container ad449cf2dd30942e213fbc895fc95bf29f36c3d22c4a139be8c02d3da6a2a855: Status 404 returned error can't find the container with id ad449cf2dd30942e213fbc895fc95bf29f36c3d22c4a139be8c02d3da6a2a855 Oct 30 02:08:18.478138 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.478092 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:08:18.478247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.478213 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:08:18.478307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.478259 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:08:18.901439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.901402 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" event={"ID":"34ef869d-3809-420e-9622-3fb670d7cf8b","Type":"ContainerStarted","Data":"f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a"} Oct 30 02:08:18.901943 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.901913 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:18.902038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.901952 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" event={"ID":"34ef869d-3809-420e-9622-3fb670d7cf8b","Type":"ContainerStarted","Data":"bfb9c98f9c9686e491a7a18ec60d8175d64fdb7d4bfe25bd468e12c702b1153e"} Oct 30 02:08:18.904175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.904111 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" event={"ID":"ecc49b69-9b46-40d2-94aa-9c76bd29f433","Type":"ContainerStarted","Data":"2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c"} Oct 30 02:08:18.904294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.904197 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" event={"ID":"ecc49b69-9b46-40d2-94aa-9c76bd29f433","Type":"ContainerStarted","Data":"ad449cf2dd30942e213fbc895fc95bf29f36c3d22c4a139be8c02d3da6a2a855"} Oct 30 02:08:18.942441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.942025 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" podStartSLOduration=2.495918351 podStartE2EDuration="2.942008634s" totalImagesPullingTime="446.090283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:18.940971715 +0000 UTC m=+2327.332779350" watchObservedRunningTime="2026-10-30 02:08:18.942008634 +0000 UTC m=+2327.333816267" Oct 30 02:08:18.992344 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:18.992287 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podStartSLOduration=1.992272469 podStartE2EDuration="1.992272469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:18.988791718 +0000 UTC m=+2327.380599353" watchObservedRunningTime="2026-10-30 02:08:18.992272469 +0000 UTC m=+2327.384080102" Oct 30 02:08:19.137256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:19.132428 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:19.163320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:19.163234 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:19.910917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:19.910884 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:19.911773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:19.911755 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:08:24.916675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:24.916641 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:08:46.711942 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:08:46.711731 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:08:54.880942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:54.880900 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:08:54.881398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:54.880971 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:56.380552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:56.380522 6275 generic.go:356] "Generic (PLEG): container finished" podID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerID="3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e" exitCode=0 Oct 30 02:08:56.381016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:56.380594 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" event={"ID":"68d1c41c-7055-41c5-9a20-3a1b41dd709b","Type":"ContainerDied","Data":"3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e"} Oct 30 02:08:56.381066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:56.381024 6275 scope.go:122] "RemoveContainer" containerID="3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e" Oct 30 02:08:56.382607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:56.382575 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:08:56.382724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:56.382663 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:08:56.382724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:56.382712 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:08:57.395090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:57.395052 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" event={"ID":"68d1c41c-7055-41c5-9a20-3a1b41dd709b","Type":"ContainerStarted","Data":"e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c"} Oct 30 02:08:57.769193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:57.769158 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:08:57.769408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:57.769357 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 02:08:57.769473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:57.769406 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 02:08:58.766916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:58.766880 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 02:08:58.767364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:58.766943 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 02:08:59.767391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:59.767355 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 02:08:59.767987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:08:59.767954 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 02:09:00.779541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:00.779507 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 02:09:00.780101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:00.779559 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 02:09:01.766525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:01.766488 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 02:09:01.766765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:01.766567 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 02:09:02.097910 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.097824 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65"] Oct 30 02:09:02.101861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.101843 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.119203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.119174 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a1028d\"" Oct 30 02:09:02.134942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.134906 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65"] Oct 30 02:09:02.187532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.187497 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gsr4\" (UniqueName: \"kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4\") pod \"payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.187717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.187542 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.332147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.331847 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gsr4\" (UniqueName: \"kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4\") pod \"payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.365189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.360952 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.365189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.331904 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.432147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.393578 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gsr4\" (UniqueName: \"kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4\") pod \"payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.432147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.423352 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:02.768760 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.768702 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 02:09:02.768982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.768784 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 02:09:02.869025 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:09:02.864943 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9d6d77_0ae5_4d23_8b2e_418e692d5dd4.slice/crio-9c1203152f6c0e758156f63d67517c69eabaa942b4f4c563dbbc050aaeb7633f WatchSource:0}: Error finding container 9c1203152f6c0e758156f63d67517c69eabaa942b4f4c563dbbc050aaeb7633f: Status 404 returned error can't find the container with id 9c1203152f6c0e758156f63d67517c69eabaa942b4f4c563dbbc050aaeb7633f Oct 30 02:09:02.895189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:02.893013 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65"] Oct 30 02:09:03.528891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.528859 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" event={"ID":"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4","Type":"ContainerStarted","Data":"8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e"} Oct 30 02:09:03.529336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.528912 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" event={"ID":"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4","Type":"ContainerStarted","Data":"9c1203152f6c0e758156f63d67517c69eabaa942b4f4c563dbbc050aaeb7633f"} Oct 30 02:09:03.529336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.528961 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:03.570830 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.570781 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" podStartSLOduration=1.570769461 podStartE2EDuration="1.570769461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:03.569722309 +0000 UTC m=+2371.961529955" watchObservedRunningTime="2026-10-30 02:09:03.570769461 +0000 UTC m=+2371.962577093" Oct 30 02:09:03.771033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.770998 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:09:03.771424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.771401 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:09:03.772212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:03.772192 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:09:14.563579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:14.563549 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:09:16.699298 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:09:16.699257 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:09:20.725353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.723987 6275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": read tcp 10.134.0.2:51270->10.134.0.184:15021: read: connection reset by peer" start-of-body= Oct 30 02:09:20.725353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.724060 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": read tcp 10.134.0.2:51270->10.134.0.184:15021: read: connection reset by peer" Oct 30 02:09:20.791928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.791889 6275 generic.go:356] "Generic (PLEG): container finished" podID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerID="2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c" exitCode=0 Oct 30 02:09:20.792093 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.791938 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" event={"ID":"ecc49b69-9b46-40d2-94aa-9c76bd29f433","Type":"ContainerDied","Data":"2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c"} Oct 30 02:09:20.792392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.792367 6275 scope.go:122] "RemoveContainer" containerID="2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c" Oct 30 02:09:20.793918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.793885 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:09:20.794000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.793953 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:09:20.794000 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:20.793985 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:09:21.808063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:21.808028 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" event={"ID":"ecc49b69-9b46-40d2-94aa-9c76bd29f433","Type":"ContainerStarted","Data":"ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874"} Oct 30 02:09:22.130348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:22.130247 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:09:22.130538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:22.130481 6275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:09:22.130597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:22.130536 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 02:09:23.129964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:23.129929 6275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:09:23.130398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:23.129989 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 02:09:24.129724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.129690 6275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:09:24.129937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.129746 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 02:09:24.363722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.363669 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl"] Oct 30 02:09:24.368579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.368557 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.401929 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.401820 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a1028d-serving-cert\"" Oct 30 02:09:24.406433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.406402 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl"] Oct 30 02:09:24.466033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.465991 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.466484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.466458 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdst\" (UniqueName: \"kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.466793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.466771 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.466955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.466938 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.571074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.571035 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdst\" (UniqueName: \"kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.596145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.594194 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.596145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.594258 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.596145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.594339 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.596743 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.596714 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.650852 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.650813 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.673280 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.652053 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.690634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.690310 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdst\" (UniqueName: \"kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst\") pod \"maas-api-e2e-autores-a1028d-64f557cb69-2tghl\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:24.983562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:24.983468 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:25.130060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:25.130021 6275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:09:25.130260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:25.130103 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 02:09:25.336814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:25.336790 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl"] Oct 30 02:09:25.836360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:25.836298 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" event={"ID":"c3db02e2-6740-4577-9277-24940634b99d","Type":"ContainerStarted","Data":"240e5c9e07f14b38d1db9ac5f38210e8e564fc7d7fbe7e2edea5d069fd6b34bd"} Oct 30 02:09:26.130486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:26.130405 6275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:09:26.130703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:26.130676 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 02:09:26.872358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:26.872246 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" event={"ID":"c3db02e2-6740-4577-9277-24940634b99d","Type":"ContainerStarted","Data":"b6018655cfeb005770e2a78e830accc3eead4f1d9220c549d8b2df6689c7bbe6"} Oct 30 02:09:26.872862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:26.872471 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:26.917293 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:26.917231 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" podStartSLOduration=2.499367409 podStartE2EDuration="2.91721021s" totalImagesPullingTime="417.842801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:26.913939805 +0000 UTC m=+2395.305747439" watchObservedRunningTime="2026-10-30 02:09:26.91721021 +0000 UTC m=+2395.309017843" Oct 30 02:09:27.288181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:27.288085 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:09:27.288961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:27.288939 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:09:27.291230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:27.291208 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:09:32.280698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:32.280667 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:09:32.294517 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:32.294495 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:09:32.886381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:32.886354 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:09:34.600923 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:34.600881 6275 scope.go:122] "RemoveContainer" containerID="04a1e312d91e3b59d2714c1998c0b3ae7a5a8a3fa3f2948fc7c76795b5011932" Oct 30 02:09:37.809020 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.808977 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk"] Oct 30 02:09:37.814619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.814592 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.843490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.843467 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"kube-root-ca.crt\"" Oct 30 02:09:37.854250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.854199 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"openshift-service-ca.crt\"" Oct 30 02:09:37.854458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.854429 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"default-dockercfg-sj8kl\"" Oct 30 02:09:37.854728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.854706 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"auth-test-model-9a1cd0-kserve-self-signed-certs\"" Oct 30 02:09:37.859959 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.859932 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk"] Oct 30 02:09:37.886941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.885655 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.886941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.885729 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.886941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.885843 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrpv\" (UniqueName: \"kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.886941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.885928 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.886941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.885963 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.886941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.885991 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.987955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.987926 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.987974 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988046 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrpv\" (UniqueName: \"kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988105 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988150 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988175 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988464 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988501 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.988618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.988573 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.990638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.990609 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:37.990747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:37.990661 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:38.013290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:38.013262 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrpv\" (UniqueName: \"kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv\") pod \"auth-test-model-9a1cd0-kserve-644b64649c-4smdk\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:38.139516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:38.139428 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:38.549436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:38.549398 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk"] Oct 30 02:09:38.549574 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:09:38.549440 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b68cbdb_8cb5_4b3b_87a9_203ae417924b.slice/crio-4db0c09f2ee7d917deb0747a81e3e78865a272fdbf8c732effefdeb0882e6bf2 WatchSource:0}: Error finding container 4db0c09f2ee7d917deb0747a81e3e78865a272fdbf8c732effefdeb0882e6bf2: Status 404 returned error can't find the container with id 4db0c09f2ee7d917deb0747a81e3e78865a272fdbf8c732effefdeb0882e6bf2 Oct 30 02:09:38.999204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:38.998254 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" event={"ID":"7b68cbdb-8cb5-4b3b-87a9-203ae417924b","Type":"ContainerStarted","Data":"2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121"} Oct 30 02:09:38.999204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:38.998307 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" event={"ID":"7b68cbdb-8cb5-4b3b-87a9-203ae417924b","Type":"ContainerStarted","Data":"4db0c09f2ee7d917deb0747a81e3e78865a272fdbf8c732effefdeb0882e6bf2"} Oct 30 02:09:39.064146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:39.053442 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" podStartSLOduration=2.053423831 podStartE2EDuration="2.053423831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:39.053384643 +0000 UTC m=+2407.445192281" watchObservedRunningTime="2026-10-30 02:09:39.053423831 +0000 UTC m=+2407.445231464" Oct 30 02:09:39.143173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:39.142355 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:44.227190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.227159 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj"] Oct 30 02:09:44.230588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.230571 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.245676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.245656 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-01a9cd-dockercfg-d7jgv\"" Oct 30 02:09:44.245676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.245657 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-01a9cd\"" Oct 30 02:09:44.260029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.260007 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj"] Oct 30 02:09:44.401207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.362439 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxgv\" (UniqueName: \"kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv\") pod \"payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.401207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.362631 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.503197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.472361 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxgv\" (UniqueName: \"kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv\") pod \"payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.503197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.472563 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.503544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.503508 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.532142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.531586 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng"] Oct 30 02:09:44.543787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.543759 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.549272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.549229 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng"] Oct 30 02:09:44.588446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.585280 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxgv\" (UniqueName: \"kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv\") pod \"payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.674979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.674940 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.675223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.675169 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j\") pod \"payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.777029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.776992 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.777275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.777181 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j\") pod \"payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.777792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.777767 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.801600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.801565 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j\") pod \"payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:44.844597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.844562 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:44.901340 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:44.901310 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:45.179683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:45.179658 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj"] Oct 30 02:09:45.181831 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:09:45.181804 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc107c565_22f4_4184_bbc3_b60038674c5f.slice/crio-d06901bcd844caf6ccf84f8c9828e3baf97ddad95916d18bc42378566fd4cce8 WatchSource:0}: Error finding container d06901bcd844caf6ccf84f8c9828e3baf97ddad95916d18bc42378566fd4cce8: Status 404 returned error can't find the container with id d06901bcd844caf6ccf84f8c9828e3baf97ddad95916d18bc42378566fd4cce8 Oct 30 02:09:45.348578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:45.348493 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng"] Oct 30 02:09:45.356836 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:09:45.356809 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a372f44_cdb5_4db0_b3c4_fe0e3e32bd4c.slice/crio-c938ba155c87a71026b7d933bed350027dd948c18b20be7338b4f61b946738de WatchSource:0}: Error finding container c938ba155c87a71026b7d933bed350027dd948c18b20be7338b4f61b946738de: Status 404 returned error can't find the container with id c938ba155c87a71026b7d933bed350027dd948c18b20be7338b4f61b946738de Oct 30 02:09:46.097201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.097161 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" event={"ID":"c107c565-22f4-4184-bbc3-b60038674c5f","Type":"ContainerStarted","Data":"25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac"} Oct 30 02:09:46.097427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.097255 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" event={"ID":"c107c565-22f4-4184-bbc3-b60038674c5f","Type":"ContainerStarted","Data":"d06901bcd844caf6ccf84f8c9828e3baf97ddad95916d18bc42378566fd4cce8"} Oct 30 02:09:46.097427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.097419 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:46.098784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.098754 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" event={"ID":"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c","Type":"ContainerStarted","Data":"92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3"} Oct 30 02:09:46.098932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.098796 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" event={"ID":"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c","Type":"ContainerStarted","Data":"c938ba155c87a71026b7d933bed350027dd948c18b20be7338b4f61b946738de"} Oct 30 02:09:46.098932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.098816 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:46.138181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.138134 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" podStartSLOduration=2.138106113 podStartE2EDuration="2.138106113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:46.134826881 +0000 UTC m=+2414.526634516" watchObservedRunningTime="2026-10-30 02:09:46.138106113 +0000 UTC m=+2414.529913745" Oct 30 02:09:46.177914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:46.177854 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" podStartSLOduration=2.177837037 podStartE2EDuration="2.177837037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:46.174581937 +0000 UTC m=+2414.566389570" watchObservedRunningTime="2026-10-30 02:09:46.177837037 +0000 UTC m=+2414.569644671" Oct 30 02:09:46.697210 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:09:46.697171 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:09:48.143402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:48.143369 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:48.159716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:48.159688 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:49.130728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:49.130696 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:09:57.109594 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:57.109560 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:09:57.110116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:57.109633 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:09:59.452743 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:59.452705 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr"] Oct 30 02:09:59.453194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:09:59.452918 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" containerID="cri-o://dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be" gracePeriod=30 Oct 30 02:10:04.747009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.746939 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 02:10:04.859866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.859831 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.859866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.859891 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.859986 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.860090 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.860177 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860416 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.860215 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860416 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.860331 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.860439 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.860631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.860486 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket\") pod \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\" (UID: \"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c\") " Oct 30 02:10:04.869101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.868940 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:04.869721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.869346 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:04.869721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.869522 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:10:04.869898 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.869881 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:04.870443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.870084 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:04.871243 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.870988 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:04.872163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.871857 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:10:04.881400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.873771 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:10:04.887643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.887615 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp" pod "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" (UID: "8efcb4b3-4afb-4420-a3d2-67017dc0ba4c"). InnerVolumeSpecName "kube-api-access-lh6xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:10:04.971683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971638 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971674 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971687 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971702 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971714 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971725 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6xp\" (UniqueName: \"kubernetes.io/projected/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-kube-api-access-lh6xp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971737 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971750 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:04.971945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:04.971761 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:10:05.251352 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.251315 6275 generic.go:356] "Generic (PLEG): container finished" podID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerID="dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be" exitCode=0 Oct 30 02:10:05.251553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.251382 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" Oct 30 02:10:05.251553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.251395 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" event={"ID":"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c","Type":"ContainerDied","Data":"dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be"} Oct 30 02:10:05.251553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.251470 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr" event={"ID":"8efcb4b3-4afb-4420-a3d2-67017dc0ba4c","Type":"ContainerDied","Data":"8b0e3ca9f4a1f2e658a00e6b71749d5c0229ccdcf86a791bd3499caa9c3ace03"} Oct 30 02:10:05.251553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.251492 6275 scope.go:122] "RemoveContainer" containerID="dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be" Oct 30 02:10:05.262224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.262206 6275 scope.go:122] "RemoveContainer" containerID="dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be" Oct 30 02:10:05.262524 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:10:05.262500 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be\": container with ID starting with dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be not found: ID does not exist" containerID="dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be" Oct 30 02:10:05.262583 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.262532 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be"} err="failed to get container status \"dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be\": rpc error: code = NotFound desc = could not find container \"dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be\": container with ID starting with dbe76bb81b15bdea11e920654b05504421643ed56f541827d9ec4f11c7c764be not found: ID does not exist" Oct 30 02:10:05.299608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.299577 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr"] Oct 30 02:10:05.310367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.310339 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fl6gr"] Oct 30 02:10:05.741779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:05.741746 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" path="/var/lib/kubelet/pods/8efcb4b3-4afb-4420-a3d2-67017dc0ba4c/volumes" Oct 30 02:10:07.373157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.373113 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r"] Oct 30 02:10:07.373886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.373865 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" Oct 30 02:10:07.373886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.373884 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" Oct 30 02:10:07.374026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.373998 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efcb4b3-4afb-4420-a3d2-67017dc0ba4c" containerName="istio-proxy" Oct 30 02:10:07.379136 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.379106 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.411059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.410033 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-01a9cd-serving-cert\"" Oct 30 02:10:07.425767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.425729 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r"] Oct 30 02:10:07.513070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.513035 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.513484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.513460 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.513597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.513580 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.513679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.513660 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4s6\" (UniqueName: \"kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.614619 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4s6\" (UniqueName: \"kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.614730 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.614951 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.614980 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.618440 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.620378 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.639211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.631839 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.674057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.673983 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4s6\" (UniqueName: \"kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6\") pod \"maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:07.704823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:07.704528 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:08.025555 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:08.025524 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r"] Oct 30 02:10:08.027819 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:10:08.027786 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb67639_18bc_4b31_8ff8_dde4cdeb81ab.slice/crio-d1a8dd71aa7cee9a9b435ec1b559a3570c3a6e2e67c9bde312cfbbc96bdbcc08 WatchSource:0}: Error finding container d1a8dd71aa7cee9a9b435ec1b559a3570c3a6e2e67c9bde312cfbbc96bdbcc08: Status 404 returned error can't find the container with id d1a8dd71aa7cee9a9b435ec1b559a3570c3a6e2e67c9bde312cfbbc96bdbcc08 Oct 30 02:10:08.275487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:08.275394 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" event={"ID":"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab","Type":"ContainerStarted","Data":"d1a8dd71aa7cee9a9b435ec1b559a3570c3a6e2e67c9bde312cfbbc96bdbcc08"} Oct 30 02:10:09.284607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:09.284566 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" event={"ID":"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab","Type":"ContainerStarted","Data":"bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26"} Oct 30 02:10:09.285018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:09.284660 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:09.369300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:09.369249 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" podStartSLOduration=1.993060464 podStartE2EDuration="2.369236034s" totalImagesPullingTime="376.17557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:09.368790459 +0000 UTC m=+2437.760598094" watchObservedRunningTime="2026-10-30 02:10:09.369236034 +0000 UTC m=+2437.761043666" Oct 30 02:10:15.297214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:10:15.297108 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:10:16.698174 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:10:16.698138 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:10:46.698880 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:10:46.698840 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:11:16.698966 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:11:16.698927 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:11:18.952928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:18.952887 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65"] Oct 30 02:11:23.953491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:23.953434 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" podUID="ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" containerName="payload-processing" containerID="cri-o://8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e" gracePeriod=25 Oct 30 02:11:24.243224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.243185 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:11:24.355027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.354997 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4\" (UniqueName: \"kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4\") pod \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " Oct 30 02:11:24.355258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.355244 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume\") pod \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\" (UID: \"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4\") " Oct 30 02:11:24.355618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.355587 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume" pod "ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" (UID: "ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:24.355741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.355707 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:24.357224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.357194 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4" pod "ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" (UID: "ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4"). InnerVolumeSpecName "kube-api-access-4gsr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:24.456991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:24.456952 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gsr4\" (UniqueName: \"kubernetes.io/projected/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4-kube-api-access-4gsr4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:25.060433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.060399 6275 generic.go:356] "Generic (PLEG): container finished" podID="ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" containerID="8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e" exitCode=0 Oct 30 02:11:25.060935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.060441 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" Oct 30 02:11:25.060935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.060480 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" event={"ID":"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4","Type":"ContainerDied","Data":"8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e"} Oct 30 02:11:25.060935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.060533 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65" event={"ID":"ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4","Type":"ContainerDied","Data":"9c1203152f6c0e758156f63d67517c69eabaa942b4f4c563dbbc050aaeb7633f"} Oct 30 02:11:25.060935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.060555 6275 scope.go:122] "RemoveContainer" containerID="8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e" Oct 30 02:11:25.072520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.072496 6275 scope.go:122] "RemoveContainer" containerID="8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e" Oct 30 02:11:25.072853 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:11:25.072832 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e\": container with ID starting with 8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e not found: ID does not exist" containerID="8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e" Oct 30 02:11:25.072916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.072863 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e"} err="failed to get container status \"8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e\": rpc error: code = NotFound desc = could not find container \"8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e\": container with ID starting with 8635053771f76597b1548aa8e090948d2ac6901b553aa59cb81d190d43f4bf7e not found: ID does not exist" Oct 30 02:11:25.110928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.110865 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65"] Oct 30 02:11:25.121440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.121408 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1028d-7f6d45d596-c5x65"] Oct 30 02:11:25.739721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:25.739685 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" path="/var/lib/kubelet/pods/ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4/volumes" Oct 30 02:11:31.988270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:31.988229 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt"] Oct 30 02:11:31.988778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:31.988502 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" containerID="cri-o://6f375b1d8630658f9129f7cf77e822598686a066d4de6b202f39529ab619b66c" gracePeriod=30 Oct 30 02:11:31.999161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:31.999111 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx"] Oct 30 02:11:31.999395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:31.999368 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" containerID="cri-o://f1e8ac61e5f9ad1d91d1d29ec09fe59b2db818af09a594d7845f9d8baf6ab166" gracePeriod=30 Oct 30 02:11:32.002617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:32.002588 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp"] Oct 30 02:11:32.002855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:32.002815 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" containerID="cri-o://67d38c7c442bc0cd42e457be00aec4ade7849e3950c8a0f971661fcb15dcfb66" gracePeriod=30 Oct 30 02:11:32.769510 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:32.769471 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl"] Oct 30 02:11:32.769739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:32.769709 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" podUID="c3db02e2-6740-4577-9277-24940634b99d" containerName="maas-api" containerID="cri-o://b6018655cfeb005770e2a78e830accc3eead4f1d9220c549d8b2df6689c7bbe6" gracePeriod=30 Oct 30 02:11:32.881772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:32.881728 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" podUID="c3db02e2-6740-4577-9277-24940634b99d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.186:8443/health\": dial tcp 10.134.0.186:8443: connect: connection refused" Oct 30 02:11:33.364295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.364255 6275 generic.go:356] "Generic (PLEG): container finished" podID="c3db02e2-6740-4577-9277-24940634b99d" containerID="b6018655cfeb005770e2a78e830accc3eead4f1d9220c549d8b2df6689c7bbe6" exitCode=0 Oct 30 02:11:33.365054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.365021 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" event={"ID":"c3db02e2-6740-4577-9277-24940634b99d","Type":"ContainerDied","Data":"b6018655cfeb005770e2a78e830accc3eead4f1d9220c549d8b2df6689c7bbe6"} Oct 30 02:11:33.394287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.394260 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:11:33.495545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.495466 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca\") pod \"c3db02e2-6740-4577-9277-24940634b99d\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " Oct 30 02:11:33.495545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.495518 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs\") pod \"c3db02e2-6740-4577-9277-24940634b99d\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " Oct 30 02:11:33.495731 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.495677 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst\" (UniqueName: \"kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst\") pod \"c3db02e2-6740-4577-9277-24940634b99d\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " Oct 30 02:11:33.495731 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.495698 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls\") pod \"c3db02e2-6740-4577-9277-24940634b99d\" (UID: \"c3db02e2-6740-4577-9277-24940634b99d\") " Oct 30 02:11:33.495839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.495812 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca" pod "c3db02e2-6740-4577-9277-24940634b99d" (UID: "c3db02e2-6740-4577-9277-24940634b99d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:33.496388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.496364 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3db02e2-6740-4577-9277-24940634b99d-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:33.497756 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.497732 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs" pod "c3db02e2-6740-4577-9277-24940634b99d" (UID: "c3db02e2-6740-4577-9277-24940634b99d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:11:33.497878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.497759 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst" pod "c3db02e2-6740-4577-9277-24940634b99d" (UID: "c3db02e2-6740-4577-9277-24940634b99d"). InnerVolumeSpecName "kube-api-access-4hdst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:33.497878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.497848 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls" pod "c3db02e2-6740-4577-9277-24940634b99d" (UID: "c3db02e2-6740-4577-9277-24940634b99d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:11:33.597380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.597346 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:33.597380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.597373 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hdst\" (UniqueName: \"kubernetes.io/projected/c3db02e2-6740-4577-9277-24940634b99d-kube-api-access-4hdst\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:33.597380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:33.597384 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3db02e2-6740-4577-9277-24940634b99d-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:34.372708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:34.372672 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" event={"ID":"c3db02e2-6740-4577-9277-24940634b99d","Type":"ContainerDied","Data":"240e5c9e07f14b38d1db9ac5f38210e8e564fc7d7fbe7e2edea5d069fd6b34bd"} Oct 30 02:11:34.372708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:34.372701 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl" Oct 30 02:11:34.373266 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:34.372743 6275 scope.go:122] "RemoveContainer" containerID="b6018655cfeb005770e2a78e830accc3eead4f1d9220c549d8b2df6689c7bbe6" Oct 30 02:11:34.419940 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:34.419906 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl"] Oct 30 02:11:34.430598 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:34.430561 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1028d-64f557cb69-2tghl"] Oct 30 02:11:35.739251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:35.739220 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3db02e2-6740-4577-9277-24940634b99d" path="/var/lib/kubelet/pods/c3db02e2-6740-4577-9277-24940634b99d/volumes" Oct 30 02:11:37.026372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.026340 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw"] Oct 30 02:11:37.027468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.027444 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" containerName="payload-processing" Oct 30 02:11:37.027602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.027469 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" containerName="payload-processing" Oct 30 02:11:37.027602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.027489 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c3db02e2-6740-4577-9277-24940634b99d" containerName="maas-api" Oct 30 02:11:37.027602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.027497 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3db02e2-6740-4577-9277-24940634b99d" containerName="maas-api" Oct 30 02:11:37.027773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.027664 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9d6d77-0ae5-4d23-8b2e-418e692d5dd4" containerName="payload-processing" Oct 30 02:11:37.027773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.027677 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3db02e2-6740-4577-9277-24940634b99d" containerName="maas-api" Oct 30 02:11:37.032257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.032232 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.053979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.053958 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-069e90-openshift-default-dockercfg-92czl\"" Oct 30 02:11:37.069695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.069671 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.069798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.069710 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.069854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.069824 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.069854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.069847 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.070022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.069998 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.070202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.070180 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.070306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.070288 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.070369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.070349 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnfp\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.070416 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.070375 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.074726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.074501 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw"] Oct 30 02:11:37.208145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.185724 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.208894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.208848 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.237485 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.229541 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.270402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.209114 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.285146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.281303 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.308361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.281545 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnfp\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.324194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.308787 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.325015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.324982 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.325256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.325238 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.325423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.325409 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.325519 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.325508 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.327884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.327838 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.328530 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.328507 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.343056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.337326 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.349744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.347756 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnfp\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.349744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.348028 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.365807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.365761 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.368226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.368158 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token\") pod \"e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.433296 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.433042 6275 generic.go:356] "Generic (PLEG): container finished" podID="a8881725-7a6b-41e2-905b-bfc750854b42" containerID="f1e8ac61e5f9ad1d91d1d29ec09fe59b2db818af09a594d7845f9d8baf6ab166" exitCode=0 Oct 30 02:11:37.433296 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.433206 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" event={"ID":"a8881725-7a6b-41e2-905b-bfc750854b42","Type":"ContainerDied","Data":"f1e8ac61e5f9ad1d91d1d29ec09fe59b2db818af09a594d7845f9d8baf6ab166"} Oct 30 02:11:37.436160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.435888 6275 generic.go:356] "Generic (PLEG): container finished" podID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerID="67d38c7c442bc0cd42e457be00aec4ade7849e3950c8a0f971661fcb15dcfb66" exitCode=0 Oct 30 02:11:37.436160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.436050 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" event={"ID":"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6","Type":"ContainerDied","Data":"67d38c7c442bc0cd42e457be00aec4ade7849e3950c8a0f971661fcb15dcfb66"} Oct 30 02:11:37.461514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.461471 6275 generic.go:356] "Generic (PLEG): container finished" podID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerID="6f375b1d8630658f9129f7cf77e822598686a066d4de6b202f39529ab619b66c" exitCode=0 Oct 30 02:11:37.462026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.461802 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" event={"ID":"e091fd66-e4b9-40d6-8687-5f4d78ea782b","Type":"ContainerDied","Data":"6f375b1d8630658f9129f7cf77e822598686a066d4de6b202f39529ab619b66c"} Oct 30 02:11:37.591271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.591116 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:37.622053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.622022 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 02:11:37.665457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.665431 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:11:37.723040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.723007 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:11:37.733704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.733675 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734181 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734228 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734263 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734290 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734324 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734361 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734417 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734441 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734482 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734550 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734588 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734611 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.734808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734646 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734669 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734696 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734712 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data\") pod \"a8881725-7a6b-41e2-905b-bfc750854b42\" (UID: \"a8881725-7a6b-41e2-905b-bfc750854b42\") " Oct 30 02:11:37.734808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.734740 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token\") pod \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\" (UID: \"e091fd66-e4b9-40d6-8687-5f4d78ea782b\") " Oct 30 02:11:37.735649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.735622 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.735887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.735860 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:37.736200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.736178 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:37.736434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.736409 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.737913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.737700 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.737913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.737812 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.739226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.738149 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.739226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.738437 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "kube-api-access-lk7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:37.739226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.738625 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.739226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.738799 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.739226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.738969 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.739226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.739105 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "kube-api-access-jhcr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:37.739613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.739487 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:11:37.739860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.739768 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.741031 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.740993 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:37.741739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.741693 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.741739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.741703 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo" pod "a8881725-7a6b-41e2-905b-bfc750854b42" (UID: "a8881725-7a6b-41e2-905b-bfc750854b42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:11:37.743294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.742856 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token" pod "e091fd66-e4b9-40d6-8687-5f4d78ea782b" (UID: "e091fd66-e4b9-40d6-8687-5f4d78ea782b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:37.835758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.835674 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.835758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.835730 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.835999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.835886 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.835999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.835966 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.835999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.835984 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.836218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.835995 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.836218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836045 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.836218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836154 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.836218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836193 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.836423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836227 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.836423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836262 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data\") pod \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\" (UID: \"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6\") " Oct 30 02:11:37.836423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836232 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:37.836423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836318 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.836624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.836587 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.837158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837019 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837090 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837212 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837233 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837248 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837272 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837288 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837319 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837337 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837351 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837367 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8881725-7a6b-41e2-905b-bfc750854b42-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837381 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhcr2\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-kube-api-access-jhcr2\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837396 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837409 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837423 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837438 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837451 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837464 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8881725-7a6b-41e2-905b-bfc750854b42-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837479 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7mn\" (UniqueName: \"kubernetes.io/projected/a8881725-7a6b-41e2-905b-bfc750854b42-kube-api-access-lk7mn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837494 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837507 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837520 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8881725-7a6b-41e2-905b-bfc750854b42-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.837652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.837533 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e091fd66-e4b9-40d6-8687-5f4d78ea782b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.838427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.838404 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:37.838598 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.838580 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:11:37.838666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.838639 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "kube-api-access-r5vp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:37.838705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.838644 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token" pod "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" (UID: "0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:37.938972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.938938 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vp4\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-kube-api-access-r5vp4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.938972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.938970 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.938972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.938980 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.939226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.938988 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:11:37.944288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.944256 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw"] Oct 30 02:11:37.947025 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:11:37.946989 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7b7182_8162_4dbe_ad1b_1376c05e3394.slice/crio-42b1fa0f51ef743a93be5a261c868821aea7ed3eab6eec1778539684aa3a045e WatchSource:0}: Error finding container 42b1fa0f51ef743a93be5a261c868821aea7ed3eab6eec1778539684aa3a045e: Status 404 returned error can't find the container with id 42b1fa0f51ef743a93be5a261c868821aea7ed3eab6eec1778539684aa3a045e Oct 30 02:11:37.949557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.949523 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:11:37.949641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.949589 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:11:37.949641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:37.949620 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:11:38.470870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.470765 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" event={"ID":"9f7b7182-8162-4dbe-ad1b-1376c05e3394","Type":"ContainerStarted","Data":"bd2346070aa12b9e0254c51780e34b3bab8de242d1d6a3161d57157232136598"} Oct 30 02:11:38.470870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.470852 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" event={"ID":"9f7b7182-8162-4dbe-ad1b-1376c05e3394","Type":"ContainerStarted","Data":"42b1fa0f51ef743a93be5a261c868821aea7ed3eab6eec1778539684aa3a045e"} Oct 30 02:11:38.472490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.472469 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" Oct 30 02:11:38.472646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.472519 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx" event={"ID":"a8881725-7a6b-41e2-905b-bfc750854b42","Type":"ContainerDied","Data":"e4ee7ed4f3b8eb046f9c24321edfd5eea3d543f89c30f508beaa44fb1e120228"} Oct 30 02:11:38.472646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.472584 6275 scope.go:122] "RemoveContainer" containerID="f1e8ac61e5f9ad1d91d1d29ec09fe59b2db818af09a594d7845f9d8baf6ab166" Oct 30 02:11:38.481158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.475171 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" Oct 30 02:11:38.481158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.475206 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp" event={"ID":"0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6","Type":"ContainerDied","Data":"5893e8ac99230a09ed4b6b4694d79ea695706b7bc8234f6e55bbfdc0489a282a"} Oct 30 02:11:38.481158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.476737 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" event={"ID":"e091fd66-e4b9-40d6-8687-5f4d78ea782b","Type":"ContainerDied","Data":"804ed95dc565c93482ab40f2629989b0d9b4b1d0deff6f49e4399246150615b6"} Oct 30 02:11:38.481158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.476795 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt" Oct 30 02:11:38.485638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.485620 6275 scope.go:122] "RemoveContainer" containerID="67d38c7c442bc0cd42e457be00aec4ade7849e3950c8a0f971661fcb15dcfb66" Oct 30 02:11:38.495219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.495200 6275 scope.go:122] "RemoveContainer" containerID="6f375b1d8630658f9129f7cf77e822598686a066d4de6b202f39529ab619b66c" Oct 30 02:11:38.520800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.520737 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" podStartSLOduration=2.520717576 podStartE2EDuration="2.520717576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:38.519175343 +0000 UTC m=+2526.910982970" watchObservedRunningTime="2026-10-30 02:11:38.520717576 +0000 UTC m=+2526.912525212" Oct 30 02:11:38.564869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.564838 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp"] Oct 30 02:11:38.570028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.570000 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vpdwp"] Oct 30 02:11:38.591995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.591970 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:38.603555 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.603535 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx"] Oct 30 02:11:38.613200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.613173 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2v7dx"] Oct 30 02:11:38.657145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.647980 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt"] Oct 30 02:11:38.702260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:38.702161 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-jwhdt"] Oct 30 02:11:39.597363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:39.597332 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:39.740884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:39.740850 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" path="/var/lib/kubelet/pods/0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6/volumes" Oct 30 02:11:39.741412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:39.741398 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" path="/var/lib/kubelet/pods/a8881725-7a6b-41e2-905b-bfc750854b42/volumes" Oct 30 02:11:39.741847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:39.741834 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" path="/var/lib/kubelet/pods/e091fd66-e4b9-40d6-8687-5f4d78ea782b/volumes" Oct 30 02:11:40.493196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:40.493157 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:40.493939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:40.493919 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:11:46.700554 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:11:46.700476 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:11:47.356244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.356196 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s"] Oct 30 02:11:47.357540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357506 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" Oct 30 02:11:47.357540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357540 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" Oct 30 02:11:47.357772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357590 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" Oct 30 02:11:47.357772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357598 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" Oct 30 02:11:47.357772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357670 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" Oct 30 02:11:47.357772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357678 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" Oct 30 02:11:47.358480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357890 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e091fd66-e4b9-40d6-8687-5f4d78ea782b" containerName="istio-proxy" Oct 30 02:11:47.358480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357916 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4bf0e9-1776-49b5-b76c-aa51d4b0d4d6" containerName="istio-proxy" Oct 30 02:11:47.358480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.357929 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8881725-7a6b-41e2-905b-bfc750854b42" containerName="istio-proxy" Oct 30 02:11:47.363656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.363637 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.381248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.381221 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s"] Oct 30 02:11:47.401311 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.401285 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-069e90\"/\"openshift-service-ca.crt\"" Oct 30 02:11:47.401491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.401474 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-069e90\"/\"noten-model-069e90-kserve-self-signed-certs\"" Oct 30 02:11:47.401575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.401550 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-069e90\"/\"kube-root-ca.crt\"" Oct 30 02:11:47.401644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.401629 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-069e90\"/\"default-dockercfg-vpj9h\"" Oct 30 02:11:47.485720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.485672 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.485879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.485817 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.485940 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.485917 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.486021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.485997 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86hj\" (UniqueName: \"kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.486098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.486084 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.486267 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.486248 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.592409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.591856 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.592409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.592011 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.592409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.592114 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.592409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.592184 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.592409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.592240 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86hj\" (UniqueName: \"kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.592409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.592318 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.633195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.604943 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.633195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.605290 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.633195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.605720 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.633195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.630554 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.644148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.644064 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.665146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.655885 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86hj\" (UniqueName: \"kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj\") pod \"noten-model-069e90-kserve-5f86d6b785-4sm5s\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:47.690144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:47.686376 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:48.125373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:48.125344 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s"] Oct 30 02:11:48.129083 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:11:48.129054 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50ceb2ab_5c87_436d_b79d_b4b535697422.slice/crio-38eeff07539e97691df73db015f2a526318a042a2fec9da5bdc591e54efedbe0 WatchSource:0}: Error finding container 38eeff07539e97691df73db015f2a526318a042a2fec9da5bdc591e54efedbe0: Status 404 returned error can't find the container with id 38eeff07539e97691df73db015f2a526318a042a2fec9da5bdc591e54efedbe0 Oct 30 02:11:48.560393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:48.559665 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" event={"ID":"50ceb2ab-5c87-436d-b79d-b4b535697422","Type":"ContainerStarted","Data":"4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0"} Oct 30 02:11:48.560393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:48.559736 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" event={"ID":"50ceb2ab-5c87-436d-b79d-b4b535697422","Type":"ContainerStarted","Data":"38eeff07539e97691df73db015f2a526318a042a2fec9da5bdc591e54efedbe0"} Oct 30 02:11:48.603205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:48.603152 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" podStartSLOduration=1.603137651 podStartE2EDuration="1.603137651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:48.599539815 +0000 UTC m=+2536.991347459" watchObservedRunningTime="2026-10-30 02:11:48.603137651 +0000 UTC m=+2536.994945281" Oct 30 02:11:48.687105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:48.687075 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:57.687543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:57.687506 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:57.699958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:57.699932 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:11:58.636666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:11:58.636631 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:12:06.415664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:06.415627 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s"] Oct 30 02:12:06.583144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:06.573998 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw"] Oct 30 02:12:06.583144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:06.574374 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" podUID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" containerName="istio-proxy" containerID="cri-o://bd2346070aa12b9e0254c51780e34b3bab8de242d1d6a3161d57157232136598" gracePeriod=30 Oct 30 02:12:10.495361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:10.495321 6275 patch_prober.go:29] interesting pod/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 02:12:10.495761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:10.495386 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" podUID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:12:11.817470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.817437 6275 generic.go:356] "Generic (PLEG): container finished" podID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" containerID="bd2346070aa12b9e0254c51780e34b3bab8de242d1d6a3161d57157232136598" exitCode=0 Oct 30 02:12:11.817811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.817507 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" event={"ID":"9f7b7182-8162-4dbe-ad1b-1376c05e3394","Type":"ContainerDied","Data":"bd2346070aa12b9e0254c51780e34b3bab8de242d1d6a3161d57157232136598"} Oct 30 02:12:11.817811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.817559 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" event={"ID":"9f7b7182-8162-4dbe-ad1b-1376c05e3394","Type":"ContainerDied","Data":"42b1fa0f51ef743a93be5a261c868821aea7ed3eab6eec1778539684aa3a045e"} Oct 30 02:12:11.817811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.817575 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b1fa0f51ef743a93be5a261c868821aea7ed3eab6eec1778539684aa3a045e" Oct 30 02:12:11.843470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.842955 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:12:11.986756 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.986672 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.986914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.986778 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.986914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.986822 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.986914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.986885 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.987033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.986955 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:11.987033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.986982 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.987115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987064 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.987209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987176 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.987272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987237 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.987332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987307 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs\") pod \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\" (UID: \"9f7b7182-8162-4dbe-ad1b-1376c05e3394\") " Oct 30 02:12:11.987332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987314 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:11.987611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987550 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:11.987611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987561 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:11.987883 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.987726 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:11.988106 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.988087 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:11.988206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.988112 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:11.988206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.988142 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:11.988206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.988155 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:11.988206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.988169 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:11.989200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.989175 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:11.989304 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.989247 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:11.989523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.989501 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:12:11.989600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:11.989583 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp" pod "9f7b7182-8162-4dbe-ad1b-1376c05e3394" (UID: "9f7b7182-8162-4dbe-ad1b-1376c05e3394"). InnerVolumeSpecName "kube-api-access-rdnfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:12.089364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.089334 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:12.089364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.089360 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:12.089364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.089369 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7b7182-8162-4dbe-ad1b-1376c05e3394-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:12.089611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.089377 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdnfp\" (UniqueName: \"kubernetes.io/projected/9f7b7182-8162-4dbe-ad1b-1376c05e3394-kube-api-access-rdnfp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:12.823345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.823313 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw" Oct 30 02:12:12.881780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.881747 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw"] Oct 30 02:12:12.892040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:12.892014 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-069e90-openshift-default-56f7865fdc-g4wvw"] Oct 30 02:12:13.742926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:13.742880 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" path="/var/lib/kubelet/pods/9f7b7182-8162-4dbe-ad1b-1376c05e3394/volumes" Oct 30 02:12:16.700708 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:16.700671 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:12:21.437773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.437725 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" podUID="50ceb2ab-5c87-436d-b79d-b4b535697422" containerName="main" containerID="cri-o://4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0" gracePeriod=45 Oct 30 02:12:21.734826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.734793 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:12:21.887675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.887640 6275 generic.go:356] "Generic (PLEG): container finished" podID="50ceb2ab-5c87-436d-b79d-b4b535697422" containerID="4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0" exitCode=0 Oct 30 02:12:21.887882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.887686 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" Oct 30 02:12:21.887882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.887724 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" event={"ID":"50ceb2ab-5c87-436d-b79d-b4b535697422","Type":"ContainerDied","Data":"4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0"} Oct 30 02:12:21.887882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.887776 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" event={"ID":"50ceb2ab-5c87-436d-b79d-b4b535697422","Type":"ContainerDied","Data":"38eeff07539e97691df73db015f2a526318a042a2fec9da5bdc591e54efedbe0"} Oct 30 02:12:21.887882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.887799 6275 scope.go:122] "RemoveContainer" containerID="4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0" Oct 30 02:12:21.896768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.896738 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs\" (UniqueName: \"kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs\") pod \"50ceb2ab-5c87-436d-b79d-b4b535697422\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " Oct 30 02:12:21.896913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.896789 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir\") pod \"50ceb2ab-5c87-436d-b79d-b4b535697422\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " Oct 30 02:12:21.896913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.896828 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache\") pod \"50ceb2ab-5c87-436d-b79d-b4b535697422\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " Oct 30 02:12:21.897050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.896935 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm\") pod \"50ceb2ab-5c87-436d-b79d-b4b535697422\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " Oct 30 02:12:21.897050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.896964 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj\" (UniqueName: \"kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj\") pod \"50ceb2ab-5c87-436d-b79d-b4b535697422\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " Oct 30 02:12:21.897205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.897064 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir" pod "50ceb2ab-5c87-436d-b79d-b4b535697422" (UID: "50ceb2ab-5c87-436d-b79d-b4b535697422"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:21.897278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.897243 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home\") pod \"50ceb2ab-5c87-436d-b79d-b4b535697422\" (UID: \"50ceb2ab-5c87-436d-b79d-b4b535697422\") " Oct 30 02:12:21.897566 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.897243 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache" pod "50ceb2ab-5c87-436d-b79d-b4b535697422" (UID: "50ceb2ab-5c87-436d-b79d-b4b535697422"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:21.897692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.897468 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home" pod "50ceb2ab-5c87-436d-b79d-b4b535697422" (UID: "50ceb2ab-5c87-436d-b79d-b4b535697422"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:21.898068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.898037 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:21.898068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.898069 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:21.898282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.898079 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:21.898472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.898422 6275 scope.go:122] "RemoveContainer" containerID="4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0" Oct 30 02:12:21.898772 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:21.898742 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0\": container with ID starting with 4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0 not found: ID does not exist" containerID="4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0" Oct 30 02:12:21.898863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.898784 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0"} err="failed to get container status \"4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0\": rpc error: code = NotFound desc = could not find container \"4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0\": container with ID starting with 4bae151fe3253483513561b5a4bdbad139411aee7f1ea1a5bd641c6f3b565bf0 not found: ID does not exist" Oct 30 02:12:21.899621 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.899595 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs" pod "50ceb2ab-5c87-436d-b79d-b4b535697422" (UID: "50ceb2ab-5c87-436d-b79d-b4b535697422"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:21.899998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.899972 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm" pod "50ceb2ab-5c87-436d-b79d-b4b535697422" (UID: "50ceb2ab-5c87-436d-b79d-b4b535697422"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:21.900966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.900943 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj" pod "50ceb2ab-5c87-436d-b79d-b4b535697422" (UID: "50ceb2ab-5c87-436d-b79d-b4b535697422"). InnerVolumeSpecName "kube-api-access-h86hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:21.999651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.999552 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ceb2ab-5c87-436d-b79d-b4b535697422-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:21.999651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.999586 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ceb2ab-5c87-436d-b79d-b4b535697422-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:21.999651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:21.999597 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86hj\" (UniqueName: \"kubernetes.io/projected/50ceb2ab-5c87-436d-b79d-b4b535697422-kube-api-access-h86hj\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:22.244038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:22.244005 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s"] Oct 30 02:12:22.248528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:22.248495 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s"] Oct 30 02:12:22.626341 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:22.626288 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-069e90/noten-model-069e90-kserve-5f86d6b785-4sm5s" podUID="50ceb2ab-5c87-436d-b79d-b4b535697422" containerName="main" probeResult="failure" output="Get \"https://10.134.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:12:23.740290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:23.740256 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ceb2ab-5c87-436d-b79d-b4b535697422" path="/var/lib/kubelet/pods/50ceb2ab-5c87-436d-b79d-b4b535697422/volumes" Oct 30 02:12:46.698492 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:46.698444 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:12:50.635617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.635577 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11187-b5249c"] Oct 30 02:12:50.636220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.636203 6275 cpu_manager.go:414] "containerMap: removing container" podUID="50ceb2ab-5c87-436d-b79d-b4b535697422" containerName="main" Oct 30 02:12:50.636220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.636219 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ceb2ab-5c87-436d-b79d-b4b535697422" containerName="main" Oct 30 02:12:50.636332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.636248 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" containerName="istio-proxy" Oct 30 02:12:50.636332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.636254 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" containerName="istio-proxy" Oct 30 02:12:50.636397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.636373 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ceb2ab-5c87-436d-b79d-b4b535697422" containerName="main" Oct 30 02:12:50.636397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.636384 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7b7182-8162-4dbe-ad1b-1376c05e3394" containerName="istio-proxy" Oct 30 02:12:50.641176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.641153 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:50.662747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.662718 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11187-b5249c"] Oct 30 02:12:50.737192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.737153 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gb4l\" (UniqueName: \"kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l\") pod \"test-curl-11187-b5249c\" (UID: \"4d3b167a-13a4-42a5-b2ad-6d70953a7b75\") " pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:50.744219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.744194 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sddb\"" Oct 30 02:12:50.838880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.838848 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gb4l\" (UniqueName: \"kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l\") pod \"test-curl-11187-b5249c\" (UID: \"4d3b167a-13a4-42a5-b2ad-6d70953a7b75\") " pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:50.863720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.863691 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gb4l\" (UniqueName: \"kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l\") pod \"test-curl-11187-b5249c\" (UID: \"4d3b167a-13a4-42a5-b2ad-6d70953a7b75\") " pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:50.955027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:50.954950 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:51.325908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:51.325882 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11187-b5249c"] Oct 30 02:12:51.328551 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:12:51.328520 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d3b167a_13a4_42a5_b2ad_6d70953a7b75.slice/crio-6af3b4b95857e044bcaf9f50ac4b96e5364184dea751e3aff852ef26a66570ff WatchSource:0}: Error finding container 6af3b4b95857e044bcaf9f50ac4b96e5364184dea751e3aff852ef26a66570ff: Status 404 returned error can't find the container with id 6af3b4b95857e044bcaf9f50ac4b96e5364184dea751e3aff852ef26a66570ff Oct 30 02:12:52.103380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:52.103345 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11187-b5249c" event={"ID":"4d3b167a-13a4-42a5-b2ad-6d70953a7b75","Type":"ContainerStarted","Data":"f3174584ec54cbe8d0e518007cc7119b3cf38195b0773a798306f33081feb599"} Oct 30 02:12:52.103380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:52.103395 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11187-b5249c" event={"ID":"4d3b167a-13a4-42a5-b2ad-6d70953a7b75","Type":"ContainerStarted","Data":"6af3b4b95857e044bcaf9f50ac4b96e5364184dea751e3aff852ef26a66570ff"} Oct 30 02:12:52.139241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:52.139190 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11187-b5249c" podStartSLOduration=2.139177916 podStartE2EDuration="2.139177916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:52.138116812 +0000 UTC m=+2600.529924445" watchObservedRunningTime="2026-10-30 02:12:52.139177916 +0000 UTC m=+2600.530985549" Oct 30 02:12:52.974264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:52.974232 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11187-b5249c"] Oct 30 02:12:52.977768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:52.977738 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11187-b5249c"] Oct 30 02:12:53.702345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.699690 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11196-456950"] Oct 30 02:12:53.702345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.701338 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" containerName="test-curl-11187-b5249c" Oct 30 02:12:53.702345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.701362 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" containerName="test-curl-11187-b5249c" Oct 30 02:12:53.702345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.701709 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" containerName="test-curl-11187-b5249c" Oct 30 02:12:53.709612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.709581 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:53.721162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.721115 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-456950"] Oct 30 02:12:53.795584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.795545 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7z92\" (UniqueName: \"kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92\") pod \"test-curl-11196-456950\" (UID: \"19403532-32c7-47c5-902b-ef2f718c933a\") " pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:53.897508 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.897471 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7z92\" (UniqueName: \"kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92\") pod \"test-curl-11196-456950\" (UID: \"19403532-32c7-47c5-902b-ef2f718c933a\") " pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:53.919831 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.919803 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7z92\" (UniqueName: \"kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92\") pod \"test-curl-11196-456950\" (UID: \"19403532-32c7-47c5-902b-ef2f718c933a\") " pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:53.954083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.953988 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488"] Oct 30 02:12:53.959732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.959706 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.984599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.984571 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3a00b8-openshift-default-dockercfg-zf4h8\"" Oct 30 02:12:53.988040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.988016 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488"] Oct 30 02:12:53.998837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.998809 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.998989 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.998871 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999042 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999147 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999262 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999311 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hcm\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999407 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999489 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999451 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:53.999549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:53.999509 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.023896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.023868 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:54.131144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.105331 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131473 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131546 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hcm\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131651 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131702 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131770 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131865 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.131925 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.132684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.132044 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.148144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.135666 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.148144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.135732 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.148144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.135998 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.150133 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.150083 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.190140 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.171859 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.192051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.191644 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.204217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.194872 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.225332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.224075 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.320183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.314975 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11187-b5249c" podUID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" containerName="test-curl-11187-b5249c" containerID="cri-o://f3174584ec54cbe8d0e518007cc7119b3cf38195b0773a798306f33081feb599" gracePeriod=2 Oct 30 02:12:54.320183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.318768 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hcm\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm\") pod \"e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.526728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.526693 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-456950"] Oct 30 02:12:54.528569 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:12:54.528541 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19403532_32c7_47c5_902b_ef2f718c933a.slice/crio-2f46eadc9aa3243d425ab459c46eef729f05ddc738e4ed2e0b43b224b04459a2 WatchSource:0}: Error finding container 2f46eadc9aa3243d425ab459c46eef729f05ddc738e4ed2e0b43b224b04459a2: Status 404 returned error can't find the container with id 2f46eadc9aa3243d425ab459c46eef729f05ddc738e4ed2e0b43b224b04459a2 Oct 30 02:12:54.575832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.575806 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:54.906468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.906382 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488"] Oct 30 02:12:54.911721 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:12:54.911687 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc52375_b571_4ab9_bba6_24ccf15db366.slice/crio-447f6fdb3526176620ce4c4f035c8d35cd8e6b3a8ba45e96425230f0e3ac6e07 WatchSource:0}: Error finding container 447f6fdb3526176620ce4c4f035c8d35cd8e6b3a8ba45e96425230f0e3ac6e07: Status 404 returned error can't find the container with id 447f6fdb3526176620ce4c4f035c8d35cd8e6b3a8ba45e96425230f0e3ac6e07 Oct 30 02:12:54.915357 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.915319 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:12:54.915473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.915394 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:12:54.915473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:54.915424 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:12:55.387146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.381845 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" event={"ID":"0cc52375-b571-4ab9-bba6-24ccf15db366","Type":"ContainerStarted","Data":"3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e"} Oct 30 02:12:55.387146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.381913 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" event={"ID":"0cc52375-b571-4ab9-bba6-24ccf15db366","Type":"ContainerStarted","Data":"447f6fdb3526176620ce4c4f035c8d35cd8e6b3a8ba45e96425230f0e3ac6e07"} Oct 30 02:12:55.390148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.389227 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-456950" event={"ID":"19403532-32c7-47c5-902b-ef2f718c933a","Type":"ContainerStarted","Data":"9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec"} Oct 30 02:12:55.390148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.389277 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-456950" event={"ID":"19403532-32c7-47c5-902b-ef2f718c933a","Type":"ContainerStarted","Data":"2f46eadc9aa3243d425ab459c46eef729f05ddc738e4ed2e0b43b224b04459a2"} Oct 30 02:12:55.428041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.427987 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podStartSLOduration=2.427973033 podStartE2EDuration="2.427973033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:55.425252931 +0000 UTC m=+2603.817060575" watchObservedRunningTime="2026-10-30 02:12:55.427973033 +0000 UTC m=+2603.819780665" Oct 30 02:12:55.461052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.460994 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11196-456950" podStartSLOduration=2.460982524 podStartE2EDuration="2.460982524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:55.460266943 +0000 UTC m=+2603.852074578" watchObservedRunningTime="2026-10-30 02:12:55.460982524 +0000 UTC m=+2603.852790157" Oct 30 02:12:55.577345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:55.577216 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:56.333869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.333840 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11196-456950"] Oct 30 02:12:56.338260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.338233 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11196-456950"] Oct 30 02:12:56.400408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.400155 6275 generic.go:356] "Generic (PLEG): container finished" podID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" containerID="f3174584ec54cbe8d0e518007cc7119b3cf38195b0773a798306f33081feb599" exitCode=137 Oct 30 02:12:56.488380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.488348 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11196-01ba21"] Oct 30 02:12:56.489604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.489577 6275 cpu_manager.go:414] "containerMap: removing container" podUID="19403532-32c7-47c5-902b-ef2f718c933a" containerName="test-curl-11196-456950" Oct 30 02:12:56.489781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.489764 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19403532-32c7-47c5-902b-ef2f718c933a" containerName="test-curl-11196-456950" Oct 30 02:12:56.490145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.490109 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="19403532-32c7-47c5-902b-ef2f718c933a" containerName="test-curl-11196-456950" Oct 30 02:12:56.507033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.507001 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:56.550316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.535606 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-01ba21"] Oct 30 02:12:56.590275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.590205 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:56.595204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.595173 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:56.635246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.635217 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msbx\" (UniqueName: \"kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx\") pod \"test-curl-11196-01ba21\" (UID: \"453a4ad0-40b7-48ab-9c26-b429c9a1f40b\") " pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:56.736593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.736564 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l\" (UniqueName: \"kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l\") pod \"4d3b167a-13a4-42a5-b2ad-6d70953a7b75\" (UID: \"4d3b167a-13a4-42a5-b2ad-6d70953a7b75\") " Oct 30 02:12:56.736925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.736903 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msbx\" (UniqueName: \"kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx\") pod \"test-curl-11196-01ba21\" (UID: \"453a4ad0-40b7-48ab-9c26-b429c9a1f40b\") " pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:56.738758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.738705 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l" pod "4d3b167a-13a4-42a5-b2ad-6d70953a7b75" (UID: "4d3b167a-13a4-42a5-b2ad-6d70953a7b75"). InnerVolumeSpecName "kube-api-access-4gb4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.758453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.758418 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msbx\" (UniqueName: \"kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx\") pod \"test-curl-11196-01ba21\" (UID: \"453a4ad0-40b7-48ab-9c26-b429c9a1f40b\") " pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:56.838216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.838181 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gb4l\" (UniqueName: \"kubernetes.io/projected/4d3b167a-13a4-42a5-b2ad-6d70953a7b75-kube-api-access-4gb4l\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.851133 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:56.851049 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:57.167677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.167654 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-01ba21"] Oct 30 02:12:57.171135 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:12:57.171091 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod453a4ad0_40b7_48ab_9c26_b429c9a1f40b.slice/crio-471e3efe649a7a79f44c986d0dd7aee467f42f8a4038032cd0aa4c831bc7f733 WatchSource:0}: Error finding container 471e3efe649a7a79f44c986d0dd7aee467f42f8a4038032cd0aa4c831bc7f733: Status 404 returned error can't find the container with id 471e3efe649a7a79f44c986d0dd7aee467f42f8a4038032cd0aa4c831bc7f733 Oct 30 02:12:57.410394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.410350 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-01ba21" event={"ID":"453a4ad0-40b7-48ab-9c26-b429c9a1f40b","Type":"ContainerStarted","Data":"a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c"} Oct 30 02:12:57.410804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.410403 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-01ba21" event={"ID":"453a4ad0-40b7-48ab-9c26-b429c9a1f40b","Type":"ContainerStarted","Data":"471e3efe649a7a79f44c986d0dd7aee467f42f8a4038032cd0aa4c831bc7f733"} Oct 30 02:12:57.411797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.411774 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:57.411935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.411843 6275 scope.go:122] "RemoveContainer" containerID="f3174584ec54cbe8d0e518007cc7119b3cf38195b0773a798306f33081feb599" Oct 30 02:12:57.412008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.411982 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11196-456950" podUID="19403532-32c7-47c5-902b-ef2f718c933a" containerName="test-curl-11196-456950" containerID="cri-o://9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec" gracePeriod=2 Oct 30 02:12:57.412291 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.412274 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:57.413457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.413437 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:12:57.449624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.449571 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11196-01ba21" podStartSLOduration=1.449556326 podStartE2EDuration="1.449556326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:57.446227416 +0000 UTC m=+2605.838035049" watchObservedRunningTime="2026-10-30 02:12:57.449556326 +0000 UTC m=+2605.841363960" Oct 30 02:12:57.501554 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:57.501507 6275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11187-b5249c\" is forbidden: User \"system:node:ip-10-0-143-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-27.ec2.internal' and this object" podUID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" pod="opendatahub/test-curl-11187-b5249c" Oct 30 02:12:57.520817 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:57.514235 6275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11196-456950\" is forbidden: User \"system:node:ip-10-0-143-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-27.ec2.internal' and this object" podUID="19403532-32c7-47c5-902b-ef2f718c933a" pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:57.741059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:57.740983 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3b167a-13a4-42a5-b2ad-6d70953a7b75" path="/var/lib/kubelet/pods/4d3b167a-13a4-42a5-b2ad-6d70953a7b75/volumes" Oct 30 02:12:58.314024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:58.313991 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11196-01ba21"] Oct 30 02:12:58.334891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:58.334860 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11196-01ba21"] Oct 30 02:12:59.429643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.429599 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11196-01ba21" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" containerName="test-curl-11196-01ba21" containerID="cri-o://a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c" gracePeriod=2 Oct 30 02:12:59.442135 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:59.442094 6275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11196-01ba21\" is forbidden: User \"system:node:ip-10-0-143-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-27.ec2.internal' and this object" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:59.635796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.635766 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-456950" Oct 30 02:12:59.647842 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:59.647800 6275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11196-01ba21\" is forbidden: User \"system:node:ip-10-0-143-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-27.ec2.internal' and this object" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:59.709146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.709031 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9"] Oct 30 02:12:59.709780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.709763 6275 cpu_manager.go:414] "containerMap: removing container" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" containerName="test-curl-11196-01ba21" Oct 30 02:12:59.709833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.709782 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" containerName="test-curl-11196-01ba21" Oct 30 02:12:59.709918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.709907 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" containerName="test-curl-11196-01ba21" Oct 30 02:12:59.713443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.713360 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:12:59.727036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.727015 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-3a00b8\"" Oct 30 02:12:59.737607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.737582 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-3a00b8-dockercfg-jgsg4\"" Oct 30 02:12:59.745391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.745363 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9"] Oct 30 02:12:59.762619 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:12:59.762583 6275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11196-01ba21\" is forbidden: User \"system:node:ip-10-0-143-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-27.ec2.internal' and this object" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:12:59.778606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.778578 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92\" (UniqueName: \"kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92\") pod \"19403532-32c7-47c5-902b-ef2f718c933a\" (UID: \"19403532-32c7-47c5-902b-ef2f718c933a\") " Oct 30 02:12:59.780861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.780832 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92" pod "19403532-32c7-47c5-902b-ef2f718c933a" (UID: "19403532-32c7-47c5-902b-ef2f718c933a"). InnerVolumeSpecName "kube-api-access-m7z92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:59.800621 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.800589 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k"] Oct 30 02:12:59.805865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.805830 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:12:59.864147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.860064 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k"] Oct 30 02:12:59.893056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.882021 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:12:59.893056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.882164 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62wqr\" (UniqueName: \"kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr\") pod \"payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:12:59.893056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.882540 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7z92\" (UniqueName: \"kubernetes.io/projected/19403532-32c7-47c5-902b-ef2f718c933a-kube-api-access-m7z92\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:00.003013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.984149 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:00.003013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.984234 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.003013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.984294 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvtw\" (UniqueName: \"kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw\") pod \"payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.003013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.984334 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62wqr\" (UniqueName: \"kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr\") pod \"payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:00.003013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:12:59.985217 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:00.012774 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.012727 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62wqr\" (UniqueName: \"kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr\") pod \"payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:00.032231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.030808 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:00.085758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.085717 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.085934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.085868 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvtw\" (UniqueName: \"kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw\") pod \"payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.086494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.086473 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.109315 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.108377 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvtw\" (UniqueName: \"kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw\") pod \"payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.278082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.278047 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:00.365327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.365299 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9"] Oct 30 02:13:00.366850 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:00.366811 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ad4b0f_1c8e_47b5_a4ee_a004016ceb8e.slice/crio-18a8cf36528f27b5ab57d82ba33615221155b9e27c2fed1b39f955656d57ee35 WatchSource:0}: Error finding container 18a8cf36528f27b5ab57d82ba33615221155b9e27c2fed1b39f955656d57ee35: Status 404 returned error can't find the container with id 18a8cf36528f27b5ab57d82ba33615221155b9e27c2fed1b39f955656d57ee35 Oct 30 02:13:00.438636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.438601 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" event={"ID":"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e","Type":"ContainerStarted","Data":"18a8cf36528f27b5ab57d82ba33615221155b9e27c2fed1b39f955656d57ee35"} Oct 30 02:13:00.441386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.441352 6275 generic.go:356] "Generic (PLEG): container finished" podID="19403532-32c7-47c5-902b-ef2f718c933a" containerID="9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec" exitCode=137 Oct 30 02:13:00.441556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.441483 6275 scope.go:122] "RemoveContainer" containerID="9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec" Oct 30 02:13:00.441642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.441626 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-456950" Oct 30 02:13:00.459412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.459391 6275 scope.go:122] "RemoveContainer" containerID="9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec" Oct 30 02:13:00.459690 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:13:00.459668 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec\": container with ID starting with 9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec not found: ID does not exist" containerID="9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec" Oct 30 02:13:00.459792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.459700 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec"} err="failed to get container status \"9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec\": rpc error: code = NotFound desc = could not find container \"9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec\": container with ID starting with 9bfa02446f1a4cb4b497d07c81442f3e12916989dbdaccb3278edfe6be2e83ec not found: ID does not exist" Oct 30 02:13:00.595292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:00.595258 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k"] Oct 30 02:13:00.597960 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:00.597928 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90615baf_1842_45de_b5ed_57f5a59d02f4.slice/crio-e9be5f46d9a7250c2787df3422e9855a84747f4405ec829c014e1b8c8066bf0b WatchSource:0}: Error finding container e9be5f46d9a7250c2787df3422e9855a84747f4405ec829c014e1b8c8066bf0b: Status 404 returned error can't find the container with id e9be5f46d9a7250c2787df3422e9855a84747f4405ec829c014e1b8c8066bf0b Oct 30 02:13:01.451022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.450989 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" event={"ID":"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e","Type":"ContainerStarted","Data":"f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832"} Oct 30 02:13:01.452447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.451080 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:01.453895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.453873 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" event={"ID":"90615baf-1842-45de-b5ed-57f5a59d02f4","Type":"ContainerStarted","Data":"cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59"} Oct 30 02:13:01.454003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.453913 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" event={"ID":"90615baf-1842-45de-b5ed-57f5a59d02f4","Type":"ContainerStarted","Data":"e9be5f46d9a7250c2787df3422e9855a84747f4405ec829c014e1b8c8066bf0b"} Oct 30 02:13:01.454003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.453932 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:01.488987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.488947 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" podStartSLOduration=2.488933224 podStartE2EDuration="2.488933224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:01.488680614 +0000 UTC m=+2609.880488246" watchObservedRunningTime="2026-10-30 02:13:01.488933224 +0000 UTC m=+2609.880740857" Oct 30 02:13:01.527718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.527669 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" podStartSLOduration=2.527654355 podStartE2EDuration="2.527654355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:01.52633357 +0000 UTC m=+2609.918141205" watchObservedRunningTime="2026-10-30 02:13:01.527654355 +0000 UTC m=+2609.919461999" Oct 30 02:13:01.615264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.615240 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:13:01.740349 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.740264 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19403532-32c7-47c5-902b-ef2f718c933a" path="/var/lib/kubelet/pods/19403532-32c7-47c5-902b-ef2f718c933a/volumes" Oct 30 02:13:01.812884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.812808 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx\" (UniqueName: \"kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx\") pod \"453a4ad0-40b7-48ab-9c26-b429c9a1f40b\" (UID: \"453a4ad0-40b7-48ab-9c26-b429c9a1f40b\") " Oct 30 02:13:01.815450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.815410 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx" pod "453a4ad0-40b7-48ab-9c26-b429c9a1f40b" (UID: "453a4ad0-40b7-48ab-9c26-b429c9a1f40b"). InnerVolumeSpecName "kube-api-access-7msbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:01.915735 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:01.915696 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7msbx\" (UniqueName: \"kubernetes.io/projected/453a4ad0-40b7-48ab-9c26-b429c9a1f40b-kube-api-access-7msbx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:02.462416 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:02.462380 6275 generic.go:356] "Generic (PLEG): container finished" podID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" containerID="a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c" exitCode=137 Oct 30 02:13:02.462950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:02.462432 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-01ba21" Oct 30 02:13:02.462950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:02.462483 6275 scope.go:122] "RemoveContainer" containerID="a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c" Oct 30 02:13:02.474345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:02.474301 6275 scope.go:122] "RemoveContainer" containerID="a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c" Oct 30 02:13:02.474630 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:13:02.474598 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c\": container with ID starting with a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c not found: ID does not exist" containerID="a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c" Oct 30 02:13:02.474757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:02.474636 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c"} err="failed to get container status \"a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c\": rpc error: code = NotFound desc = could not find container \"a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c\": container with ID starting with a2f554c022a36e60eb9308ee6a2ed82c93188a34175f6bd5581c1acf1f8da51c not found: ID does not exist" Oct 30 02:13:03.741542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:03.741511 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453a4ad0-40b7-48ab-9c26-b429c9a1f40b" path="/var/lib/kubelet/pods/453a4ad0-40b7-48ab-9c26-b429c9a1f40b/volumes" Oct 30 02:13:11.990633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:11.990546 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64"] Oct 30 02:13:11.996331 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:11.996308 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.010955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.010929 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-58fad4\"" Oct 30 02:13:12.021373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.021347 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-58fad4-dockercfg-xppfk\"" Oct 30 02:13:12.026663 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.026639 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64"] Oct 30 02:13:12.253511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.250034 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw"] Oct 30 02:13:12.256988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.256106 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw"] Oct 30 02:13:12.256988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.256234 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.307169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.307111 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrrl\" (UniqueName: \"kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl\") pod \"payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.307335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.307296 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4jf\" (UniqueName: \"kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf\") pod \"payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.307335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.307329 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.307516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.307485 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.409947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.409906 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrrl\" (UniqueName: \"kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl\") pod \"payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.410175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.410013 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4jf\" (UniqueName: \"kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf\") pod \"payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.410175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.410038 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.410175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.410139 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.411057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.411031 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.411276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.411249 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.432210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.432183 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4jf\" (UniqueName: \"kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf\") pod \"payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.432341 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.432273 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrrl\" (UniqueName: \"kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl\") pod \"payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.466270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.466239 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:13:12.466742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.466718 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:13:12.589103 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.589066 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:12.610786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.610757 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:12.912545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.912516 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw"] Oct 30 02:13:12.914530 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:12.914494 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eae7013_7f1b_4066_babf_ed2ee657212a.slice/crio-b6ae8319f2a1d7e117cf16d14e0965efe5b261ee3139c76bef721ba3fdc0651b WatchSource:0}: Error finding container b6ae8319f2a1d7e117cf16d14e0965efe5b261ee3139c76bef721ba3fdc0651b: Status 404 returned error can't find the container with id b6ae8319f2a1d7e117cf16d14e0965efe5b261ee3139c76bef721ba3fdc0651b Oct 30 02:13:12.945815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:12.945789 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64"] Oct 30 02:13:12.963637 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:12.963609 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b82dfb5_2853_4663_8c19_c83c5d32ec0b.slice/crio-23bb851ea3661efeca11dcda7450864a970d5b8f633fc214c7db2fe1dfca176f WatchSource:0}: Error finding container 23bb851ea3661efeca11dcda7450864a970d5b8f633fc214c7db2fe1dfca176f: Status 404 returned error can't find the container with id 23bb851ea3661efeca11dcda7450864a970d5b8f633fc214c7db2fe1dfca176f Oct 30 02:13:13.649236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.649197 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" event={"ID":"1b82dfb5-2853-4663-8c19-c83c5d32ec0b","Type":"ContainerStarted","Data":"f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab"} Oct 30 02:13:13.649708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.649250 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" event={"ID":"1b82dfb5-2853-4663-8c19-c83c5d32ec0b","Type":"ContainerStarted","Data":"23bb851ea3661efeca11dcda7450864a970d5b8f633fc214c7db2fe1dfca176f"} Oct 30 02:13:13.649708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.649399 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:13.651052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.651027 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" event={"ID":"7eae7013-7f1b-4066-babf-ed2ee657212a","Type":"ContainerStarted","Data":"8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1"} Oct 30 02:13:13.651166 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.651068 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" event={"ID":"7eae7013-7f1b-4066-babf-ed2ee657212a","Type":"ContainerStarted","Data":"b6ae8319f2a1d7e117cf16d14e0965efe5b261ee3139c76bef721ba3fdc0651b"} Oct 30 02:13:13.651166 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.651116 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:13.689486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.689423 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" podStartSLOduration=2.689406773 podStartE2EDuration="2.689406773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:13.686888099 +0000 UTC m=+2622.078695741" watchObservedRunningTime="2026-10-30 02:13:13.689406773 +0000 UTC m=+2622.081214406" Oct 30 02:13:13.726146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:13.726059 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" podStartSLOduration=1.726043859 podStartE2EDuration="1.726043859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:13.723975755 +0000 UTC m=+2622.115783390" watchObservedRunningTime="2026-10-30 02:13:13.726043859 +0000 UTC m=+2622.117851491" Oct 30 02:13:14.549143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:14.543863 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht"] Oct 30 02:13:14.549143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:14.544113 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" containerID="cri-o://211e79c73180356922536452775ec9b97d3f59588434f921d4d5e81337eb636f" gracePeriod=30 Oct 30 02:13:14.567018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:14.566185 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5"] Oct 30 02:13:14.567018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:14.566575 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" containerID="cri-o://fe9c2de58d4c10024b83bf283aee34a976d124c3a4fd9838be6ad8fe31262083" gracePeriod=30 Oct 30 02:13:14.572193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:14.571868 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4"] Oct 30 02:13:14.572193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:14.572079 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" containerID="cri-o://d21e6b4853be94efdb0f2a00428a384203a6bb155cb84dcb0c469283dacf0f85" gracePeriod=30 Oct 30 02:13:16.698802 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:13:16.698762 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:13:19.705747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.705550 6275 generic.go:356] "Generic (PLEG): container finished" podID="58b61c35-b18f-443a-aa16-ff4678888a49" containerID="d21e6b4853be94efdb0f2a00428a384203a6bb155cb84dcb0c469283dacf0f85" exitCode=0 Oct 30 02:13:19.705747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.705652 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" event={"ID":"58b61c35-b18f-443a-aa16-ff4678888a49","Type":"ContainerDied","Data":"d21e6b4853be94efdb0f2a00428a384203a6bb155cb84dcb0c469283dacf0f85"} Oct 30 02:13:19.707395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.707367 6275 generic.go:356] "Generic (PLEG): container finished" podID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerID="fe9c2de58d4c10024b83bf283aee34a976d124c3a4fd9838be6ad8fe31262083" exitCode=0 Oct 30 02:13:19.707561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.707454 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" event={"ID":"052c0793-2f29-41d8-8afa-9ca36b98789e","Type":"ContainerDied","Data":"fe9c2de58d4c10024b83bf283aee34a976d124c3a4fd9838be6ad8fe31262083"} Oct 30 02:13:19.709075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.708957 6275 generic.go:356] "Generic (PLEG): container finished" podID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerID="211e79c73180356922536452775ec9b97d3f59588434f921d4d5e81337eb636f" exitCode=0 Oct 30 02:13:19.709075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.709032 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" event={"ID":"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee","Type":"ContainerDied","Data":"211e79c73180356922536452775ec9b97d3f59588434f921d4d5e81337eb636f"} Oct 30 02:13:19.866331 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.866259 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:13:19.883758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.883710 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.883939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.883784 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.883939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.883863 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.884075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.883944 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.884075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.884024 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.884223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.884170 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.884582 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.884548 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.884820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.884794 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.885015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.884994 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.890222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.890187 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.890337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.890323 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.890408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.890361 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data\") pod \"052c0793-2f29-41d8-8afa-9ca36b98789e\" (UID: \"052c0793-2f29-41d8-8afa-9ca36b98789e\") " Oct 30 02:13:19.890895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.890865 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:19.891067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.891047 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.891261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.891244 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052c0793-2f29-41d8-8afa-9ca36b98789e-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.891261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.891261 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.891408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.891270 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.891408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.891277 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.891408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.891285 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.892627 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.892283 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:19.892627 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.892356 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.895383 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.895362 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:19.897414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.897391 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn" pod "052c0793-2f29-41d8-8afa-9ca36b98789e" (UID: "052c0793-2f29-41d8-8afa-9ca36b98789e"). InnerVolumeSpecName "kube-api-access-2vvvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:19.920051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.920024 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:13:19.937213 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.937190 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:13:19.991815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.991770 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.991997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.991827 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.991997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.991874 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.991997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.991968 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992081 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992115 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992186 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992240 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992278 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992319 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992361 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992399 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992435 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992478 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992571 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992600 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:19.992783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992651 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token\") pod \"58b61c35-b18f-443a-aa16-ff4678888a49\" (UID: \"58b61c35-b18f-443a-aa16-ff4678888a49\") " Oct 30 02:13:19.993617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992847 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.993617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.992876 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:19.993617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.993220 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.993617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.993283 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.993617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.993391 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket\") pod \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\" (UID: \"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee\") " Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.993901 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994043 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994084 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994243 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994266 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994278 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994290 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994305 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994319 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvvn\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-kube-api-access-2vvvn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994334 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994347 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052c0793-2f29-41d8-8afa-9ca36b98789e-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994360 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994371 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b61c35-b18f-443a-aa16-ff4678888a49-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994386 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994269 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994282 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.994433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.994383 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.998173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.998147 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "kube-api-access-49p2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:19.998173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.998280 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:19.998173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.998324 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.998173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.998416 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:19.998173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.998445 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:19.999888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:19.999858 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr" pod "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" (UID: "90a6b4ea-7cae-4540-8ca8-ac9912fd70ee"). InnerVolumeSpecName "kube-api-access-dm7tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:20.000296 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.000273 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:20.000391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.000371 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo" pod "58b61c35-b18f-443a-aa16-ff4678888a49" (UID: "58b61c35-b18f-443a-aa16-ff4678888a49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:20.095537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095499 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095530 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095539 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7tr\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-kube-api-access-dm7tr\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095549 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095559 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095567 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b61c35-b18f-443a-aa16-ff4678888a49-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095574 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095583 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49p2w\" (UniqueName: \"kubernetes.io/projected/58b61c35-b18f-443a-aa16-ff4678888a49-kube-api-access-49p2w\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095591 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b61c35-b18f-443a-aa16-ff4678888a49-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095599 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.095764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.095606 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:20.718112 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.718081 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" Oct 30 02:13:20.718670 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.718115 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4" event={"ID":"58b61c35-b18f-443a-aa16-ff4678888a49","Type":"ContainerDied","Data":"5b69938da55459e9838c4b055ee9e858a33b8e7cc2282c08e8d97702190df373"} Oct 30 02:13:20.718670 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.718208 6275 scope.go:122] "RemoveContainer" containerID="d21e6b4853be94efdb0f2a00428a384203a6bb155cb84dcb0c469283dacf0f85" Oct 30 02:13:20.719916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.719890 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" Oct 30 02:13:20.719916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.719906 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5" event={"ID":"052c0793-2f29-41d8-8afa-9ca36b98789e","Type":"ContainerDied","Data":"f915f48682bd3c821fdc03c944a818e716a40cc010a43d85c74ccb4eb5ebf72e"} Oct 30 02:13:20.721963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.721933 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" event={"ID":"90a6b4ea-7cae-4540-8ca8-ac9912fd70ee","Type":"ContainerDied","Data":"87d822d93d97ab6dce7df5f777c28a1097ddb98a86ea44c16b5ac8f878dbe625"} Oct 30 02:13:20.722131 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.722035 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht" Oct 30 02:13:20.735054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.735034 6275 scope.go:122] "RemoveContainer" containerID="fe9c2de58d4c10024b83bf283aee34a976d124c3a4fd9838be6ad8fe31262083" Oct 30 02:13:20.746028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.746000 6275 scope.go:122] "RemoveContainer" containerID="211e79c73180356922536452775ec9b97d3f59588434f921d4d5e81337eb636f" Oct 30 02:13:20.747427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.747407 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv"] Oct 30 02:13:20.748150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748107 6275 cpu_manager.go:414] "containerMap: removing container" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" Oct 30 02:13:20.748150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748134 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" Oct 30 02:13:20.748237 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748172 6275 cpu_manager.go:414] "containerMap: removing container" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" Oct 30 02:13:20.748237 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748178 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" Oct 30 02:13:20.748237 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748198 6275 cpu_manager.go:414] "containerMap: removing container" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" Oct 30 02:13:20.748237 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748204 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" Oct 30 02:13:20.748369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748303 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" containerName="istio-proxy" Oct 30 02:13:20.748369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748318 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" containerName="istio-proxy" Oct 30 02:13:20.748369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.748330 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" containerName="istio-proxy" Oct 30 02:13:20.754839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.754806 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.771752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.771428 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-3a00b8-serving-cert\"" Oct 30 02:13:20.777739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.777706 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv"] Oct 30 02:13:20.796997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.796963 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4"] Oct 30 02:13:20.801862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.801831 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-hxpx4"] Oct 30 02:13:20.805213 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.805182 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.805338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.805224 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.805338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.805320 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.805513 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.805494 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhhr8\" (UniqueName: \"kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.860461 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.860382 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5"] Oct 30 02:13:20.863477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.863449 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s46k5"] Oct 30 02:13:20.907746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.907507 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.907746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.907563 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.907746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.907609 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.907746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.907710 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhhr8\" (UniqueName: \"kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.909018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.908990 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.910751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.910723 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.910963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.910943 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.929318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.929291 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhhr8\" (UniqueName: \"kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8\") pod \"maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:20.942845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.942814 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht"] Oct 30 02:13:20.945928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:20.945903 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-8gkht"] Oct 30 02:13:21.082139 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.082088 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:21.402168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.402066 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv"] Oct 30 02:13:21.405209 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:21.405178 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c28a557_34b1_4cf2_8e2e_938dfd6b6857.slice/crio-30074e306b5c82cdf073bfdc805ad1a1a8f50babcd486d70165b6b411df9fed2 WatchSource:0}: Error finding container 30074e306b5c82cdf073bfdc805ad1a1a8f50babcd486d70165b6b411df9fed2: Status 404 returned error can't find the container with id 30074e306b5c82cdf073bfdc805ad1a1a8f50babcd486d70165b6b411df9fed2 Oct 30 02:13:21.406480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.406462 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:13:21.729597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.729507 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" event={"ID":"5c28a557-34b1-4cf2-8e2e-938dfd6b6857","Type":"ContainerStarted","Data":"30074e306b5c82cdf073bfdc805ad1a1a8f50babcd486d70165b6b411df9fed2"} Oct 30 02:13:21.741296 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.741261 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052c0793-2f29-41d8-8afa-9ca36b98789e" path="/var/lib/kubelet/pods/052c0793-2f29-41d8-8afa-9ca36b98789e/volumes" Oct 30 02:13:21.741838 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.741812 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b61c35-b18f-443a-aa16-ff4678888a49" path="/var/lib/kubelet/pods/58b61c35-b18f-443a-aa16-ff4678888a49/volumes" Oct 30 02:13:21.742395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:21.742378 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a6b4ea-7cae-4540-8ca8-ac9912fd70ee" path="/var/lib/kubelet/pods/90a6b4ea-7cae-4540-8ca8-ac9912fd70ee/volumes" Oct 30 02:13:22.331079 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:22.331043 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk"] Oct 30 02:13:22.741987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:22.741843 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" event={"ID":"5c28a557-34b1-4cf2-8e2e-938dfd6b6857","Type":"ContainerStarted","Data":"04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f"} Oct 30 02:13:22.741987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:22.741899 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:22.785570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:22.785388 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" podStartSLOduration=2.260076586 podStartE2EDuration="2.785371391s" totalImagesPullingTime="525.294805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:22.782450484 +0000 UTC m=+2631.174258117" watchObservedRunningTime="2026-10-30 02:13:22.785371391 +0000 UTC m=+2631.177179064" Oct 30 02:13:24.670244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:24.670214 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:13:24.670802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:24.670288 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:13:28.758410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:28.758381 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:13:30.179323 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.179288 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf"] Oct 30 02:13:30.183517 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.183495 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.201596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.201574 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a0fc73\"" Oct 30 02:13:30.201711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.201675 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a0fc73-dockercfg-gq7s2\"" Oct 30 02:13:30.221166 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.215775 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf"] Oct 30 02:13:30.391604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.391558 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.391981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.391954 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hls7d\" (UniqueName: \"kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d\") pod \"payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.432358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.432266 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk"] Oct 30 02:13:30.442745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.442708 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk"] Oct 30 02:13:30.442933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.442821 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.493723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.493691 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.493889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.493738 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc89v\" (UniqueName: \"kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v\") pod \"payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.493889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.493821 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.493889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.493855 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hls7d\" (UniqueName: \"kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d\") pod \"payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.494467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.494448 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.515526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.515498 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hls7d\" (UniqueName: \"kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d\") pod \"payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:30.595182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.595114 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.595334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.595198 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc89v\" (UniqueName: \"kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v\") pod \"payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.595782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.595763 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.616778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.616752 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc89v\" (UniqueName: \"kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v\") pod \"payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.770866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.770836 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:30.797702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:30.797660 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:31.087257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.087231 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk"] Oct 30 02:13:31.089512 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:31.089482 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7413499b_ba8b_4489_bc32_2655b26ab793.slice/crio-88741ae5ee663d960d6d5a9f9065067f06293f81a9b9bc3ef71475ca2a783626 WatchSource:0}: Error finding container 88741ae5ee663d960d6d5a9f9065067f06293f81a9b9bc3ef71475ca2a783626: Status 404 returned error can't find the container with id 88741ae5ee663d960d6d5a9f9065067f06293f81a9b9bc3ef71475ca2a783626 Oct 30 02:13:31.123288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.123264 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf"] Oct 30 02:13:31.132555 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:31.132526 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369285b9_8086_43a0_a3df_36de41f1ea34.slice/crio-a5123cdcf0719785ab6c19626d6bb5b7c171ae7f54391ce0f29eda8da03d6b16 WatchSource:0}: Error finding container a5123cdcf0719785ab6c19626d6bb5b7c171ae7f54391ce0f29eda8da03d6b16: Status 404 returned error can't find the container with id a5123cdcf0719785ab6c19626d6bb5b7c171ae7f54391ce0f29eda8da03d6b16 Oct 30 02:13:31.810647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.810607 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" event={"ID":"7413499b-ba8b-4489-bc32-2655b26ab793","Type":"ContainerStarted","Data":"13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363"} Oct 30 02:13:31.811107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.810665 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" event={"ID":"7413499b-ba8b-4489-bc32-2655b26ab793","Type":"ContainerStarted","Data":"88741ae5ee663d960d6d5a9f9065067f06293f81a9b9bc3ef71475ca2a783626"} Oct 30 02:13:31.811107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.810687 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:31.812949 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.812915 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" event={"ID":"369285b9-8086-43a0-a3df-36de41f1ea34","Type":"ContainerStarted","Data":"6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6"} Oct 30 02:13:31.813092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.812972 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" event={"ID":"369285b9-8086-43a0-a3df-36de41f1ea34","Type":"ContainerStarted","Data":"a5123cdcf0719785ab6c19626d6bb5b7c171ae7f54391ce0f29eda8da03d6b16"} Oct 30 02:13:31.813092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.813001 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:31.848713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.848664 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" podStartSLOduration=1.84865047 podStartE2EDuration="1.84865047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:31.847615914 +0000 UTC m=+2640.239423548" watchObservedRunningTime="2026-10-30 02:13:31.84865047 +0000 UTC m=+2640.240458139" Oct 30 02:13:31.888006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:31.887952 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" podStartSLOduration=1.887938636 podStartE2EDuration="1.887938636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:31.884781859 +0000 UTC m=+2640.276589491" watchObservedRunningTime="2026-10-30 02:13:31.887938636 +0000 UTC m=+2640.279746308" Oct 30 02:13:34.341473 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.341440 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr"] Oct 30 02:13:34.348180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.348153 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.371072 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.371027 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-58fad4-serving-cert\"" Oct 30 02:13:34.374792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.374765 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr"] Oct 30 02:13:34.463475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.463437 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.463695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.463632 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.463781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.463748 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.463835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.463820 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgt5m\" (UniqueName: \"kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.566849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.565236 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.566849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.565474 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.566849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.565576 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.566849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.565654 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgt5m\" (UniqueName: \"kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.567966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.567937 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.583144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.575831 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.588432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.588401 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.594230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.594109 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgt5m\" (UniqueName: \"kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m\") pod \"maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.665407 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.665374 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:34.975306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:34.975169 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr"] Oct 30 02:13:34.983051 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:34.983015 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428a6289_6cc8_4c54_8103_9833110bb7c3.slice/crio-31fa19956d3a1050af39c4a333f40367b42e9fd9d24ac65c7dfc2a23f39032be WatchSource:0}: Error finding container 31fa19956d3a1050af39c4a333f40367b42e9fd9d24ac65c7dfc2a23f39032be: Status 404 returned error can't find the container with id 31fa19956d3a1050af39c4a333f40367b42e9fd9d24ac65c7dfc2a23f39032be Oct 30 02:13:35.840346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:35.840308 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" event={"ID":"428a6289-6cc8-4c54-8103-9833110bb7c3","Type":"ContainerStarted","Data":"04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242"} Oct 30 02:13:35.840797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:35.840367 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" event={"ID":"428a6289-6cc8-4c54-8103-9833110bb7c3","Type":"ContainerStarted","Data":"31fa19956d3a1050af39c4a333f40367b42e9fd9d24ac65c7dfc2a23f39032be"} Oct 30 02:13:35.840797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:35.840396 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:35.881199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:35.881074 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" podStartSLOduration=1.545705816 podStartE2EDuration="1.881056476s" totalImagesPullingTime="335.35066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:35.879804901 +0000 UTC m=+2644.271612547" watchObservedRunningTime="2026-10-30 02:13:35.881056476 +0000 UTC m=+2644.272864105" Oct 30 02:13:37.354622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.354587 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" podUID="7b68cbdb-8cb5-4b3b-87a9-203ae417924b" containerName="main" containerID="cri-o://2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121" gracePeriod=45 Oct 30 02:13:37.647580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.647552 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:13:37.767725 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.767694 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home\") pod \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " Oct 30 02:13:37.767914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.767784 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm\") pod \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " Oct 30 02:13:37.767914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.767856 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs\") pod \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " Oct 30 02:13:37.767914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.767904 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir\") pod \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " Oct 30 02:13:37.768087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.767956 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache\") pod \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " Oct 30 02:13:37.768087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.768017 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv\" (UniqueName: \"kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv\") pod \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\" (UID: \"7b68cbdb-8cb5-4b3b-87a9-203ae417924b\") " Oct 30 02:13:37.768087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.767952 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home" pod "7b68cbdb-8cb5-4b3b-87a9-203ae417924b" (UID: "7b68cbdb-8cb5-4b3b-87a9-203ae417924b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:37.768288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.768171 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir" pod "7b68cbdb-8cb5-4b3b-87a9-203ae417924b" (UID: "7b68cbdb-8cb5-4b3b-87a9-203ae417924b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:37.768288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.768234 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache" pod "7b68cbdb-8cb5-4b3b-87a9-203ae417924b" (UID: "7b68cbdb-8cb5-4b3b-87a9-203ae417924b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:37.768644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.768623 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:37.768644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.768646 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:37.768796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.768655 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:37.770454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.770428 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv" pod "7b68cbdb-8cb5-4b3b-87a9-203ae417924b" (UID: "7b68cbdb-8cb5-4b3b-87a9-203ae417924b"). InnerVolumeSpecName "kube-api-access-nwrpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:37.770554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.770460 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm" pod "7b68cbdb-8cb5-4b3b-87a9-203ae417924b" (UID: "7b68cbdb-8cb5-4b3b-87a9-203ae417924b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:37.770601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.770566 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs" pod "7b68cbdb-8cb5-4b3b-87a9-203ae417924b" (UID: "7b68cbdb-8cb5-4b3b-87a9-203ae417924b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:37.876197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.875607 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:37.876197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.875647 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrpv\" (UniqueName: \"kubernetes.io/projected/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-kube-api-access-nwrpv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:37.876197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.875663 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b68cbdb-8cb5-4b3b-87a9-203ae417924b-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:13:37.887387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.887352 6275 generic.go:356] "Generic (PLEG): container finished" podID="7b68cbdb-8cb5-4b3b-87a9-203ae417924b" containerID="2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121" exitCode=0 Oct 30 02:13:37.887591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.887401 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" Oct 30 02:13:37.887591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.887438 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" event={"ID":"7b68cbdb-8cb5-4b3b-87a9-203ae417924b","Type":"ContainerDied","Data":"2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121"} Oct 30 02:13:37.887591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.887482 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk" event={"ID":"7b68cbdb-8cb5-4b3b-87a9-203ae417924b","Type":"ContainerDied","Data":"4db0c09f2ee7d917deb0747a81e3e78865a272fdbf8c732effefdeb0882e6bf2"} Oct 30 02:13:37.887591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.887499 6275 scope.go:122] "RemoveContainer" containerID="2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121" Oct 30 02:13:37.899157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.899108 6275 scope.go:122] "RemoveContainer" containerID="2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121" Oct 30 02:13:37.899450 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:13:37.899428 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121\": container with ID starting with 2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121 not found: ID does not exist" containerID="2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121" Oct 30 02:13:37.899502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.899458 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121"} err="failed to get container status \"2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121\": rpc error: code = NotFound desc = could not find container \"2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121\": container with ID starting with 2473a7da602543ee2fc5d1cf515956d4ec219c1726ca7b87df01df2d639da121 not found: ID does not exist" Oct 30 02:13:37.934619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.934591 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk"] Oct 30 02:13:37.937840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:37.937816 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/auth-test-model-9a1cd0-kserve-644b64649c-4smdk"] Oct 30 02:13:39.739971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:39.739935 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b68cbdb-8cb5-4b3b-87a9-203ae417924b" path="/var/lib/kubelet/pods/7b68cbdb-8cb5-4b3b-87a9-203ae417924b/volumes" Oct 30 02:13:41.868173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:41.867968 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:13:42.823393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:42.823364 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:13:42.823599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:42.823430 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:13:46.726898 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:13:46.726854 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:13:51.084630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:51.084555 6275 generic.go:356] "Generic (PLEG): container finished" podID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerID="3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e" exitCode=0 Oct 30 02:13:51.084630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:51.084624 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" event={"ID":"0cc52375-b571-4ab9-bba6-24ccf15db366","Type":"ContainerDied","Data":"3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e"} Oct 30 02:13:51.085071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:51.085060 6275 scope.go:122] "RemoveContainer" containerID="3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e" Oct 30 02:13:51.086749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:51.086716 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:13:51.086856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:51.086786 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:13:51.086856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:51.086817 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:13:52.093429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:52.093392 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" event={"ID":"0cc52375-b571-4ab9-bba6-24ccf15db366","Type":"ContainerStarted","Data":"4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a"} Oct 30 02:13:52.577770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:52.577741 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:13:52.578517 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:52.578487 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 02:13:52.578767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:52.578725 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 02:13:53.583146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:53.582791 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 02:13:53.583146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:53.582875 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 02:13:54.576440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:54.576397 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 02:13:54.576637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:54.576464 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 02:13:55.192918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.192885 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw"] Oct 30 02:13:55.193705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.193675 6275 cpu_manager.go:414] "containerMap: removing container" podUID="7b68cbdb-8cb5-4b3b-87a9-203ae417924b" containerName="main" Oct 30 02:13:55.193705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.193699 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68cbdb-8cb5-4b3b-87a9-203ae417924b" containerName="main" Oct 30 02:13:55.193882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.193870 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68cbdb-8cb5-4b3b-87a9-203ae417924b" containerName="main" Oct 30 02:13:55.196652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.196625 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.210230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.210209 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a0fc73-serving-cert\"" Oct 30 02:13:55.219097 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.219070 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw"] Oct 30 02:13:55.304686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.304655 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.304856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.304777 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.304856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.304826 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjtdm\" (UniqueName: \"kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.304940 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.304865 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.406366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.406332 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjtdm\" (UniqueName: \"kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.406578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.406394 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.406655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.406609 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.406722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.406691 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.407350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.407323 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.409182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.409159 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.409305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.409194 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.430818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.430786 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjtdm\" (UniqueName: \"kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm\") pod \"maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.510213 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.509885 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:55.576677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.576634 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 02:13:55.576881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.576716 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 02:13:55.882769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:55.882735 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw"] Oct 30 02:13:55.885235 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:55.885204 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37349c06_fce3_45a9_9128_344fa1508a06.slice/crio-c5907da282d11d24a5fdbd615942d8352eb5da3b5151a5b1d37dbc80b87db806 WatchSource:0}: Error finding container c5907da282d11d24a5fdbd615942d8352eb5da3b5151a5b1d37dbc80b87db806: Status 404 returned error can't find the container with id c5907da282d11d24a5fdbd615942d8352eb5da3b5151a5b1d37dbc80b87db806 Oct 30 02:13:56.127967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:56.127908 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" event={"ID":"37349c06-fce3-45a9-9128-344fa1508a06","Type":"ContainerStarted","Data":"c5907da282d11d24a5fdbd615942d8352eb5da3b5151a5b1d37dbc80b87db806"} Oct 30 02:13:56.576694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:56.576606 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 02:13:56.576694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:56.576662 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 02:13:57.136569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:57.136522 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" event={"ID":"37349c06-fce3-45a9-9128-344fa1508a06","Type":"ContainerStarted","Data":"d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8"} Oct 30 02:13:57.136776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:57.136598 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:13:57.188330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:57.188272 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" podStartSLOduration=1.791963284 podStartE2EDuration="2.188258528s" totalImagesPullingTime="396.295244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:57.187407896 +0000 UTC m=+2665.579215542" watchObservedRunningTime="2026-10-30 02:13:57.188258528 +0000 UTC m=+2665.580066161" Oct 30 02:13:57.412291 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:57.412212 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:13:57.582064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:57.582024 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 02:13:57.587213 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:57.582643 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 02:13:58.581597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.581566 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:13:58.582897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.582872 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:13:58.709984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.709946 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n"] Oct 30 02:13:58.714401 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.714371 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:58.737727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.737697 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6b8bfa-dockercfg-hz7wn\"" Oct 30 02:13:58.738011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.737758 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6b8bfa\"" Oct 30 02:13:58.743829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.743752 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n"] Oct 30 02:13:58.799180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.799094 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5wx\" (UniqueName: \"kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx\") pod \"payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:58.799392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.799305 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:58.886200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.885237 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9"] Oct 30 02:13:58.922165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.908040 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5wx\" (UniqueName: \"kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx\") pod \"payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:58.922165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.908201 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:58.923355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.923320 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:58.930149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.924661 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9"] Oct 30 02:13:58.930454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.925296 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:58.951252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:58.951078 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5wx\" (UniqueName: \"kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx\") pod \"payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:59.040377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.040341 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:13:59.112686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.111919 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnc5\" (UniqueName: \"kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5\") pod \"payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.112686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.111988 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.214592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.214504 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnc5\" (UniqueName: \"kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5\") pod \"payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.214592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.214555 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.215363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.215340 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.236321 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.236290 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnc5\" (UniqueName: \"kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5\") pod \"payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.295290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.295253 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:13:59.392832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.392794 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n"] Oct 30 02:13:59.641103 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:13:59.641068 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9"] Oct 30 02:13:59.645255 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:13:59.645223 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ccdb02_96b3_4b75_822c_e69ddfa45793.slice/crio-fbdbb99af9513ec1b76952587a064323963265d07eea5891cf247ab330dd329d WatchSource:0}: Error finding container fbdbb99af9513ec1b76952587a064323963265d07eea5891cf247ab330dd329d: Status 404 returned error can't find the container with id fbdbb99af9513ec1b76952587a064323963265d07eea5891cf247ab330dd329d Oct 30 02:14:00.175935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.175903 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" event={"ID":"310a8418-7f15-4283-b7b7-ebf83096d94c","Type":"ContainerStarted","Data":"ae555c9d5eb4bdd57a3f53b4463db00b0ac707db9fba2ec3dd887997eacd7839"} Oct 30 02:14:00.176176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.175952 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" event={"ID":"310a8418-7f15-4283-b7b7-ebf83096d94c","Type":"ContainerStarted","Data":"b9fc30c1c4551deeeaad69639fc47ca75ef00f551a8a30d3f006a4a849c5bc39"} Oct 30 02:14:00.176176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.176012 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:14:00.177780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.177753 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" event={"ID":"d8ccdb02-96b3-4b75-822c-e69ddfa45793","Type":"ContainerStarted","Data":"553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8"} Oct 30 02:14:00.177894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.177799 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" event={"ID":"d8ccdb02-96b3-4b75-822c-e69ddfa45793","Type":"ContainerStarted","Data":"fbdbb99af9513ec1b76952587a064323963265d07eea5891cf247ab330dd329d"} Oct 30 02:14:00.177894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.177826 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:14:00.215965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.215918 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" podStartSLOduration=2.215904758 podStartE2EDuration="2.215904758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:00.212719993 +0000 UTC m=+2668.604527627" watchObservedRunningTime="2026-10-30 02:14:00.215904758 +0000 UTC m=+2668.607712390" Oct 30 02:14:00.252897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:00.252852 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" podStartSLOduration=2.252838333 podStartE2EDuration="2.252838333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:00.250286257 +0000 UTC m=+2668.642093889" watchObservedRunningTime="2026-10-30 02:14:00.252838333 +0000 UTC m=+2668.644645966" Oct 30 02:14:03.148217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:03.148192 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:14:06.284051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:06.284012 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs"] Oct 30 02:14:06.284568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:06.284308 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" containerID="cri-o://774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c" gracePeriod=30 Oct 30 02:14:11.187719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.187676 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:14:11.188338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.187762 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:14:11.682844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.682816 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:14:11.825782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.825740 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.826084 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.826062 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.826258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.826238 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.826602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.826583 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.826744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.826729 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.826854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.826841 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.826965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.826950 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.827088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.827074 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.827226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.827211 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data\") pod \"304167dd-cf30-452f-bfce-b5c7bb64ed10\" (UID: \"304167dd-cf30-452f-bfce-b5c7bb64ed10\") " Oct 30 02:14:11.828300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.828280 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.828749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.828731 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.829150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.829114 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:11.905934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.905872 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.906928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.906890 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.926181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.924437 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:11.934742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.934503 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.934742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.934540 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304167dd-cf30-452f-bfce-b5c7bb64ed10-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.934742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.934555 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.934742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.934572 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.934742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.934589 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.934742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.934603 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.940309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.940274 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "kube-api-access-gjl4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:11.941262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.941232 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.942202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:11.941605 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token" pod "304167dd-cf30-452f-bfce-b5c7bb64ed10" (UID: "304167dd-cf30-452f-bfce-b5c7bb64ed10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:12.036203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.036168 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:12.036203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.036205 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:12.036420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.036220 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjl4x\" (UniqueName: \"kubernetes.io/projected/304167dd-cf30-452f-bfce-b5c7bb64ed10-kube-api-access-gjl4x\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:12.267892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.267846 6275 generic.go:356] "Generic (PLEG): container finished" podID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerID="774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c" exitCode=0 Oct 30 02:14:12.268338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.267901 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" Oct 30 02:14:12.268338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.267921 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" event={"ID":"304167dd-cf30-452f-bfce-b5c7bb64ed10","Type":"ContainerDied","Data":"774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c"} Oct 30 02:14:12.268338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.267995 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs" event={"ID":"304167dd-cf30-452f-bfce-b5c7bb64ed10","Type":"ContainerDied","Data":"cfb6115a54d946db1c19e1dcd340576099edfc000e48844f38b9deba06efce2c"} Oct 30 02:14:12.268338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.268022 6275 scope.go:122] "RemoveContainer" containerID="774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c" Oct 30 02:14:12.281460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.281440 6275 scope.go:122] "RemoveContainer" containerID="774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c" Oct 30 02:14:12.281761 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:14:12.281737 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c\": container with ID starting with 774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c not found: ID does not exist" containerID="774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c" Oct 30 02:14:12.281857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.281765 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c"} err="failed to get container status \"774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c\": rpc error: code = NotFound desc = could not find container \"774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c\": container with ID starting with 774fa3147a81e955c93b4d8c4a283f70c7bf3d14a4246ff683fed5eb2044092c not found: ID does not exist" Oct 30 02:14:12.319159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.319130 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs"] Oct 30 02:14:12.331052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:12.331015 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-29mqs"] Oct 30 02:14:13.740408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:13.740374 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" path="/var/lib/kubelet/pods/304167dd-cf30-452f-bfce-b5c7bb64ed10/volumes" Oct 30 02:14:16.698466 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:14:16.698425 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:14:19.853728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.853692 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs"] Oct 30 02:14:19.854492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.854473 6275 cpu_manager.go:414] "containerMap: removing container" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" Oct 30 02:14:19.854560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.854495 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" Oct 30 02:14:19.854629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.854620 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="304167dd-cf30-452f-bfce-b5c7bb64ed10" containerName="istio-proxy" Oct 30 02:14:19.858618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.858597 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:19.883761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.883731 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-93ec46\"" Oct 30 02:14:19.883943 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.883856 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-93ec46-dockercfg-x58r8\"" Oct 30 02:14:19.888785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.888759 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs"] Oct 30 02:14:19.993147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.991978 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clm47\" (UniqueName: \"kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47\") pod \"payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:19.993147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:19.992053 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:20.100409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:20.100054 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clm47\" (UniqueName: \"kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47\") pod \"payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:20.114187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:20.114058 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:20.114187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:20.114145 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:20.129798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:20.129765 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clm47\" (UniqueName: \"kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47\") pod \"payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:20.172301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:20.172267 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:20.479369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:20.479289 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs"] Oct 30 02:14:20.483210 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:14:20.483179 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b095cab_6798_4fcc_960d_fdf54d7e6522.slice/crio-c5b5d02dccce137af122201eb2165be78aaa7179baf57cd9bd5ee40fc2dedba6 WatchSource:0}: Error finding container c5b5d02dccce137af122201eb2165be78aaa7179baf57cd9bd5ee40fc2dedba6: Status 404 returned error can't find the container with id c5b5d02dccce137af122201eb2165be78aaa7179baf57cd9bd5ee40fc2dedba6 Oct 30 02:14:21.388942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:21.388905 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" event={"ID":"4b095cab-6798-4fcc-960d-fdf54d7e6522","Type":"ContainerStarted","Data":"1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719"} Oct 30 02:14:21.389402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:21.388971 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" event={"ID":"4b095cab-6798-4fcc-960d-fdf54d7e6522","Type":"ContainerStarted","Data":"c5b5d02dccce137af122201eb2165be78aaa7179baf57cd9bd5ee40fc2dedba6"} Oct 30 02:14:21.389402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:21.388999 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:21.429706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:21.429643 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" podStartSLOduration=2.429627917 podStartE2EDuration="2.429627917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:21.426507226 +0000 UTC m=+2689.818314860" watchObservedRunningTime="2026-10-30 02:14:21.429627917 +0000 UTC m=+2689.821435549" Oct 30 02:14:28.983622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:28.983585 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr"] Oct 30 02:14:28.984220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:28.983778 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" containerID="cri-o://d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979" gracePeriod=30 Oct 30 02:14:28.994509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:28.994480 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2"] Oct 30 02:14:28.994742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:28.994709 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" containerID="cri-o://e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67" gracePeriod=30 Oct 30 02:14:32.192515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:32.192476 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:32.192963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:32.192539 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:32.333649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:32.333621 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:14:32.346927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:32.346900 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:14:32.399437 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:32.399406 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:14:34.151653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.151615 6275 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 02:14:34.152042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.151667 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 02:14:34.299702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.299676 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 02:14:34.314450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.314429 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 02:14:34.382185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.381136 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.382989 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.382961 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.383306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.383287 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.383501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.383485 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.384836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.384813 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.385058 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385042 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.385254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385240 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.385364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385353 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.385464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385453 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.385558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385546 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.385664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385653 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.385796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385785 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.385964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.385951 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.386077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386064 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.386932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386284 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.386932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386334 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.386932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386375 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert\") pod \"6b15538e-985f-47e5-9ebf-f4762969d4b3\" (UID: \"6b15538e-985f-47e5-9ebf-f4762969d4b3\") " Oct 30 02:14:34.386932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386417 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l\") pod \"b2859bee-9ad3-4267-96fb-7ea985b726be\" (UID: \"b2859bee-9ad3-4267-96fb-7ea985b726be\") " Oct 30 02:14:34.386932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386421 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.386932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.386594 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.387334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.387076 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.387334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.387095 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.387803 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.387776 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.399457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.390590 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:34.400186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.400061 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:34.403588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.402401 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.403588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.402598 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.405664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.405410 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.405664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.405500 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:34.405863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.405749 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.406021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.405935 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.419438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.419395 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:34.420048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.419839 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.420048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.419857 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:34.420048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.419933 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "kube-api-access-vmt7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:34.420048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.419948 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "kube-api-access-gbrzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:34.420048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.419965 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token" pod "6b15538e-985f-47e5-9ebf-f4762969d4b3" (UID: "6b15538e-985f-47e5-9ebf-f4762969d4b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:34.420847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.420824 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token" pod "b2859bee-9ad3-4267-96fb-7ea985b726be" (UID: "b2859bee-9ad3-4267-96fb-7ea985b726be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:34.488766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488732 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488758 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488768 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488777 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488785 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488792 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488799 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488806 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488815 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488823 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b15538e-985f-47e5-9ebf-f4762969d4b3-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488831 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmt7l\" (UniqueName: \"kubernetes.io/projected/b2859bee-9ad3-4267-96fb-7ea985b726be-kube-api-access-vmt7l\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488840 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2859bee-9ad3-4267-96fb-7ea985b726be-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488848 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrzb\" (UniqueName: \"kubernetes.io/projected/6b15538e-985f-47e5-9ebf-f4762969d4b3-kube-api-access-gbrzb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488856 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488864 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b15538e-985f-47e5-9ebf-f4762969d4b3-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.488991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.488872 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2859bee-9ad3-4267-96fb-7ea985b726be-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:14:34.509700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.509668 6275 generic.go:356] "Generic (PLEG): container finished" podID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerID="d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979" exitCode=0 Oct 30 02:14:34.509837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.509715 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" Oct 30 02:14:34.509837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.509752 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" event={"ID":"b2859bee-9ad3-4267-96fb-7ea985b726be","Type":"ContainerDied","Data":"d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979"} Oct 30 02:14:34.509837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.509825 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr" event={"ID":"b2859bee-9ad3-4267-96fb-7ea985b726be","Type":"ContainerDied","Data":"790870cbafff561a0c8884a3302cc59471c334d580eb8508bc00009841081b38"} Oct 30 02:14:34.509967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.509847 6275 scope.go:122] "RemoveContainer" containerID="d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979" Oct 30 02:14:34.511854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.511829 6275 generic.go:356] "Generic (PLEG): container finished" podID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerID="e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67" exitCode=0 Oct 30 02:14:34.511991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.511874 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" Oct 30 02:14:34.511991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.511876 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" event={"ID":"6b15538e-985f-47e5-9ebf-f4762969d4b3","Type":"ContainerDied","Data":"e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67"} Oct 30 02:14:34.511991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.511920 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2" event={"ID":"6b15538e-985f-47e5-9ebf-f4762969d4b3","Type":"ContainerDied","Data":"8129a3c911e61c25b967d2acc2acc20399bf15688ea8631dc82c56dff91549be"} Oct 30 02:14:34.525173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.523836 6275 scope.go:122] "RemoveContainer" containerID="d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979" Oct 30 02:14:34.525173 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:14:34.524390 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979\": container with ID starting with d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979 not found: ID does not exist" containerID="d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979" Oct 30 02:14:34.525173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.524418 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979"} err="failed to get container status \"d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979\": rpc error: code = NotFound desc = could not find container \"d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979\": container with ID starting with d307410cf5e28d143d74a17430e34dd3f09e782e895e1020f1e94ee111185979 not found: ID does not exist" Oct 30 02:14:34.525173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.524439 6275 scope.go:122] "RemoveContainer" containerID="e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67" Oct 30 02:14:34.539425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.539408 6275 scope.go:122] "RemoveContainer" containerID="cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330" Oct 30 02:14:34.548946 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.548924 6275 scope.go:122] "RemoveContainer" containerID="e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67" Oct 30 02:14:34.549272 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:14:34.549244 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67\": container with ID starting with e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67 not found: ID does not exist" containerID="e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67" Oct 30 02:14:34.549396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.549278 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67"} err="failed to get container status \"e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67\": rpc error: code = NotFound desc = could not find container \"e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67\": container with ID starting with e05b53008566286e28cdb8ba7a97705133d9ebaaeee9f0784f48922eb45bfd67 not found: ID does not exist" Oct 30 02:14:34.549396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.549300 6275 scope.go:122] "RemoveContainer" containerID="cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330" Oct 30 02:14:34.549565 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:14:34.549542 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330\": container with ID starting with cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330 not found: ID does not exist" containerID="cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330" Oct 30 02:14:34.549631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.549571 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330"} err="failed to get container status \"cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330\": rpc error: code = NotFound desc = could not find container \"cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330\": container with ID starting with cf19646753ecd52c14beddb49fc7f38f0287906697309a7e2f5616a0a0933330 not found: ID does not exist" Oct 30 02:14:34.573319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.573291 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2"] Oct 30 02:14:34.578679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.578656 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-btgq2"] Oct 30 02:14:34.608658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.608633 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr"] Oct 30 02:14:34.613786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:34.613761 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-2vdsr"] Oct 30 02:14:35.740033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:35.739999 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" path="/var/lib/kubelet/pods/6b15538e-985f-47e5-9ebf-f4762969d4b3/volumes" Oct 30 02:14:35.740590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:35.740574 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" path="/var/lib/kubelet/pods/b2859bee-9ad3-4267-96fb-7ea985b726be/volumes" Oct 30 02:14:37.026747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.026713 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz"] Oct 30 02:14:37.027484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027467 6275 cpu_manager.go:414] "containerMap: removing container" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" Oct 30 02:14:37.027542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027486 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" Oct 30 02:14:37.027542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027497 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" Oct 30 02:14:37.027542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027502 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" Oct 30 02:14:37.027542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027517 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" Oct 30 02:14:37.027542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027523 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" Oct 30 02:14:37.027702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027627 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2859bee-9ad3-4267-96fb-7ea985b726be" containerName="istio-proxy" Oct 30 02:14:37.027702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027636 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" Oct 30 02:14:37.027702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.027643 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b15538e-985f-47e5-9ebf-f4762969d4b3" containerName="istio-proxy" Oct 30 02:14:37.035369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.035338 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.053082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.053053 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz"] Oct 30 02:14:37.059908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.059885 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0fdc4e\"" Oct 30 02:14:37.060048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.059885 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0fdc4e-dockercfg-pm8dx\"" Oct 30 02:14:37.173642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.158463 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.173642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.171271 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkjw\" (UniqueName: \"kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw\") pod \"payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.193173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.180513 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k"] Oct 30 02:14:37.246286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.243661 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k"] Oct 30 02:14:37.246286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.243760 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.272275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.272246 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.272521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.272494 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkjw\" (UniqueName: \"kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw\") pod \"payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.273606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.273575 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.337486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.337410 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkjw\" (UniqueName: \"kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw\") pod \"payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.350302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.350274 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:37.374471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.374441 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mjm\" (UniqueName: \"kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm\") pod \"payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.374623 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.374573 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.476607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.476566 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mjm\" (UniqueName: \"kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm\") pod \"payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.476778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.476725 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.477721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.477681 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.499679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.499646 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mjm\" (UniqueName: \"kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm\") pod \"payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.569619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.569591 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:37.662664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.662487 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz"] Oct 30 02:14:37.665870 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:14:37.665841 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ce7a1c_cc4e_4576_b408_a3ea63a88c1e.slice/crio-7fae047eb939ced6091dd9c462015effeaf7425f7359b16d21c22e929b9f80f6 WatchSource:0}: Error finding container 7fae047eb939ced6091dd9c462015effeaf7425f7359b16d21c22e929b9f80f6: Status 404 returned error can't find the container with id 7fae047eb939ced6091dd9c462015effeaf7425f7359b16d21c22e929b9f80f6 Oct 30 02:14:37.888627 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:37.888599 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k"] Oct 30 02:14:37.890878 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:14:37.890851 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode530cbf4_a38c_4a03_a242_948d62ece7d7.slice/crio-08d03aaf658765a57cc152996b5d5c46b293c3fc7252d524a9de0012bf55f043 WatchSource:0}: Error finding container 08d03aaf658765a57cc152996b5d5c46b293c3fc7252d524a9de0012bf55f043: Status 404 returned error can't find the container with id 08d03aaf658765a57cc152996b5d5c46b293c3fc7252d524a9de0012bf55f043 Oct 30 02:14:38.541757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.541714 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" event={"ID":"e530cbf4-a38c-4a03-a242-948d62ece7d7","Type":"ContainerStarted","Data":"b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108"} Oct 30 02:14:38.542293 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.541780 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" event={"ID":"e530cbf4-a38c-4a03-a242-948d62ece7d7","Type":"ContainerStarted","Data":"08d03aaf658765a57cc152996b5d5c46b293c3fc7252d524a9de0012bf55f043"} Oct 30 02:14:38.542293 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.541951 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:38.543610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.543582 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" event={"ID":"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e","Type":"ContainerStarted","Data":"734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90"} Oct 30 02:14:38.543742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.543637 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:38.543742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.543661 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" event={"ID":"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e","Type":"ContainerStarted","Data":"7fae047eb939ced6091dd9c462015effeaf7425f7359b16d21c22e929b9f80f6"} Oct 30 02:14:38.581428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.581355 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" podStartSLOduration=1.5813389820000001 podStartE2EDuration="1.581338982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:38.579474176 +0000 UTC m=+2706.971281812" watchObservedRunningTime="2026-10-30 02:14:38.581338982 +0000 UTC m=+2706.973146615" Oct 30 02:14:38.627249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:38.627196 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" podStartSLOduration=1.627181649 podStartE2EDuration="1.627181649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:38.623594476 +0000 UTC m=+2707.015402109" watchObservedRunningTime="2026-10-30 02:14:38.627181649 +0000 UTC m=+2707.018989282" Oct 30 02:14:46.697702 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:14:46.697608 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:14:48.933505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:48.933468 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl"] Oct 30 02:14:48.937051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:48.937032 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:48.961865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:48.961826 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-93ec46-serving-cert\"" Oct 30 02:14:48.965833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:48.965803 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl"] Oct 30 02:14:49.047333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.047299 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.047509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.047478 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmhs\" (UniqueName: \"kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.047554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.047510 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.047620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.047595 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.153196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.152146 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmhs\" (UniqueName: \"kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.153196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.152220 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.153196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.152293 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.153196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.152375 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.153557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.153362 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.164197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.164168 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.166950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.166926 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.177616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.177590 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmhs\" (UniqueName: \"kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs\") pod \"maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.252072 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.251979 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:49.553770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.553742 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:14:49.553966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.553808 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:14:49.563307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.563261 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl"] Oct 30 02:14:49.564096 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:14:49.564057 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600da90f_44d7_45ec_8dab_41133df0f8cc.slice/crio-fa53fcc5a559d8284452562bb2a94de7dbf85df7df5b7c2a189e9af24dc5c10c WatchSource:0}: Error finding container fa53fcc5a559d8284452562bb2a94de7dbf85df7df5b7c2a189e9af24dc5c10c: Status 404 returned error can't find the container with id fa53fcc5a559d8284452562bb2a94de7dbf85df7df5b7c2a189e9af24dc5c10c Oct 30 02:14:49.640807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:49.640772 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" event={"ID":"600da90f-44d7-45ec-8dab-41133df0f8cc","Type":"ContainerStarted","Data":"fa53fcc5a559d8284452562bb2a94de7dbf85df7df5b7c2a189e9af24dc5c10c"} Oct 30 02:14:50.649528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:50.649495 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" event={"ID":"600da90f-44d7-45ec-8dab-41133df0f8cc","Type":"ContainerStarted","Data":"96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f"} Oct 30 02:14:50.649975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:50.649639 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:14:50.690749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:50.690692 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" podStartSLOduration=2.346823022 podStartE2EDuration="2.690677587s" totalImagesPullingTime="343.854565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:50.689368792 +0000 UTC m=+2719.081176430" watchObservedRunningTime="2026-10-30 02:14:50.690677587 +0000 UTC m=+2719.082485218" Oct 30 02:14:56.663876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:14:56.663839 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:15:00.243358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.242676 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf"] Oct 30 02:15:00.247780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.247738 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.276907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.276871 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf"] Oct 30 02:15:00.295317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.295282 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:15:00.354880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.354833 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7f6\" (UniqueName: \"kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.355214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.355106 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.457024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.456979 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.457240 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.457217 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7f6\" (UniqueName: \"kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.457909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.457887 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.479305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.479266 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7f6\" (UniqueName: \"kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.581013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.580978 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:00.886440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:00.886398 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf"] Oct 30 02:15:00.887658 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:15:00.887632 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b668be_f4da_431d_b394_a5aa8cef16c3.slice/crio-ac03180456322d94f944e9ca2a9c6cf54728a07fff109c4715e63f4c419c276d WatchSource:0}: Error finding container ac03180456322d94f944e9ca2a9c6cf54728a07fff109c4715e63f4c419c276d: Status 404 returned error can't find the container with id ac03180456322d94f944e9ca2a9c6cf54728a07fff109c4715e63f4c419c276d Oct 30 02:15:01.751202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:01.751164 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerStarted","Data":"ac03180456322d94f944e9ca2a9c6cf54728a07fff109c4715e63f4c419c276d"} Oct 30 02:15:03.769519 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:03.769487 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerStarted","Data":"01c9ed5427170c0830c444296a91ce304b22c2ad9897735b40f2512827c3b490"} Oct 30 02:15:03.807412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:03.807359 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" podStartSLOduration=1.3120766640000001 podStartE2EDuration="3.807345321s" totalImagesPullingTime="2.495268657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.805079046 +0000 UTC m=+2732.196886679" watchObservedRunningTime="2026-10-30 02:15:03.807345321 +0000 UTC m=+2732.199152980" Oct 30 02:15:04.396477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.396443 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp"] Oct 30 02:15:04.400654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.400623 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.417737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.417710 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9dfb3c-openshift-default-dockercfg-r8wg4\"" Oct 30 02:15:04.431983 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.431946 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp"] Oct 30 02:15:04.570147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.544036 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.617675 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.617814 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.617850 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.618015 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.618057 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlf9\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.618091 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618621 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.618230 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.618621 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.618276 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719241 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719289 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719414 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719529 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719570 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719753 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719809 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlf9\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719843 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.719902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719877 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.720023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719982 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.720116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.719983 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.720116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.720094 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.720265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.720166 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.720265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.720186 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.721832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.721804 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.722272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.722252 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.741815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.741791 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.741933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.741837 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlf9\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:04.779660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.779623 6275 generic.go:356] "Generic (PLEG): container finished" podID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerID="01c9ed5427170c0830c444296a91ce304b22c2ad9897735b40f2512827c3b490" exitCode=60 Oct 30 02:15:04.780075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.779707 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerDied","Data":"01c9ed5427170c0830c444296a91ce304b22c2ad9897735b40f2512827c3b490"} Oct 30 02:15:04.780075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:04.779991 6275 scope.go:122] "RemoveContainer" containerID="01c9ed5427170c0830c444296a91ce304b22c2ad9897735b40f2512827c3b490" Oct 30 02:15:05.017840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.017797 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:05.377851 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.377821 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp"] Oct 30 02:15:05.379767 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:15:05.379738 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c06325f_3df6_48c3_8f02_53aeef8a2de2.slice/crio-c2e7c5e035bf90ad7148fea802ca8cbd0a07e0ca71cc41821f45a566bd0e298e WatchSource:0}: Error finding container c2e7c5e035bf90ad7148fea802ca8cbd0a07e0ca71cc41821f45a566bd0e298e: Status 404 returned error can't find the container with id c2e7c5e035bf90ad7148fea802ca8cbd0a07e0ca71cc41821f45a566bd0e298e Oct 30 02:15:05.382516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.382486 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:15:05.382585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.382548 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:15:05.382626 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.382587 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:15:05.789865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.789828 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerStarted","Data":"e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4"} Oct 30 02:15:05.791831 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.791806 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" event={"ID":"5c06325f-3df6-48c3-8f02-53aeef8a2de2","Type":"ContainerStarted","Data":"49c562bea926f8a407dc2a17b91ebf66400f7d0259fee0ea4dd2fa8074a08dfa"} Oct 30 02:15:05.791987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.791848 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" event={"ID":"5c06325f-3df6-48c3-8f02-53aeef8a2de2","Type":"ContainerStarted","Data":"c2e7c5e035bf90ad7148fea802ca8cbd0a07e0ca71cc41821f45a566bd0e298e"} Oct 30 02:15:05.883232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:05.883165 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podStartSLOduration=1.883150257 podStartE2EDuration="1.883150257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:05.881453413 +0000 UTC m=+2734.273261045" watchObservedRunningTime="2026-10-30 02:15:05.883150257 +0000 UTC m=+2734.274957890" Oct 30 02:15:06.018691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.018658 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:06.024867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.024843 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:06.803283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.803241 6275 generic.go:356] "Generic (PLEG): container finished" podID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerID="e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4" exitCode=60 Oct 30 02:15:06.803752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.803330 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerDied","Data":"e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4"} Oct 30 02:15:06.803752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.803390 6275 scope.go:122] "RemoveContainer" containerID="01c9ed5427170c0830c444296a91ce304b22c2ad9897735b40f2512827c3b490" Oct 30 02:15:06.803752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.803628 6275 scope.go:122] "RemoveContainer" containerID="e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4" Oct 30 02:15:06.803927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.803895 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:06.804175 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:15:06.803993 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf_odh-ai-gateway-infra(61b668be-f4da-431d-b394-a5aa8cef16c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" Oct 30 02:15:06.804898 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:06.804877 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:15:07.812769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:07.812734 6275 scope.go:122] "RemoveContainer" containerID="e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4" Oct 30 02:15:07.813250 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:15:07.813110 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf_odh-ai-gateway-infra(61b668be-f4da-431d-b394-a5aa8cef16c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" Oct 30 02:15:11.148141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.143996 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462"] Oct 30 02:15:11.205372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.205339 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.208947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.208916 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462"] Oct 30 02:15:11.249753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.249724 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9dfb3c\"" Oct 30 02:15:11.250446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.250427 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9dfb3c-dockercfg-9lv46\"" Oct 30 02:15:11.353604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.353570 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r\") pod \"payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.353805 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.353751 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.455652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.455553 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r\") pod \"payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.455840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.455682 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.456486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.456460 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.478511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.478476 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r\") pod \"payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.584094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.584054 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:11.899738 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:11.899713 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462"] Oct 30 02:15:11.903332 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:15:11.903303 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6fe799_f177_4de5_b3d1_be217af32a1e.slice/crio-72ed956221037e69f10277d8dbdafbabdd17a09de889db13acdc0bb2a81d5732 WatchSource:0}: Error finding container 72ed956221037e69f10277d8dbdafbabdd17a09de889db13acdc0bb2a81d5732: Status 404 returned error can't find the container with id 72ed956221037e69f10277d8dbdafbabdd17a09de889db13acdc0bb2a81d5732 Oct 30 02:15:12.851140 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:12.851081 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" event={"ID":"6d6fe799-f177-4de5-b3d1-be217af32a1e","Type":"ContainerStarted","Data":"e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577"} Oct 30 02:15:12.851601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:12.851187 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" event={"ID":"6d6fe799-f177-4de5-b3d1-be217af32a1e","Type":"ContainerStarted","Data":"72ed956221037e69f10277d8dbdafbabdd17a09de889db13acdc0bb2a81d5732"} Oct 30 02:15:12.851601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:12.851371 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:12.893549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:12.893459 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" podStartSLOduration=1.8934446249999999 podStartE2EDuration="1.893444625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:12.890981127 +0000 UTC m=+2741.282788772" watchObservedRunningTime="2026-10-30 02:15:12.893444625 +0000 UTC m=+2741.285252257" Oct 30 02:15:14.368074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.368039 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs"] Oct 30 02:15:14.373087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.373065 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.390501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.390476 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0ce950\"" Oct 30 02:15:14.401245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.401222 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0ce950-dockercfg-mcdts\"" Oct 30 02:15:14.406327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.406290 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs"] Oct 30 02:15:14.509147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.509091 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh"] Oct 30 02:15:14.541146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.534615 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.580228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.579510 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh"] Oct 30 02:15:14.632157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.629218 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.632157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.629292 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzfl\" (UniqueName: \"kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl\") pod \"payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.731094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.731058 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.731274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.731178 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.731274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.731227 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khr95\" (UniqueName: \"kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95\") pod \"payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.731274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.731259 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzfl\" (UniqueName: \"kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl\") pod \"payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.732008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.731980 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.753560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.753538 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzfl\" (UniqueName: \"kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl\") pod \"payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:14.832236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.832187 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.832430 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.832277 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khr95\" (UniqueName: \"kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95\") pod \"payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.833080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.833056 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.854174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.854137 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khr95\" (UniqueName: \"kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95\") pod \"payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.889896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.889825 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:14.989738 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:14.988468 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:15.207683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.207652 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh"] Oct 30 02:15:15.209755 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:15:15.209730 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fc8538_c30f_4242_9342_ed8661810e6a.slice/crio-db5ffda18fd6aa712b8d63c6b810a4c6a8a785090858e12580bcb49ad49f5dea WatchSource:0}: Error finding container db5ffda18fd6aa712b8d63c6b810a4c6a8a785090858e12580bcb49ad49f5dea: Status 404 returned error can't find the container with id db5ffda18fd6aa712b8d63c6b810a4c6a8a785090858e12580bcb49ad49f5dea Oct 30 02:15:15.327378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.327337 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs"] Oct 30 02:15:15.352313 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:15:15.352275 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3e07b5_6ab5_4e0a_ba02_4f50786777e0.slice/crio-446e10db009b58e0517e8ac4cda761af510ff31d0322c3799093b7725e6e3c9d WatchSource:0}: Error finding container 446e10db009b58e0517e8ac4cda761af510ff31d0322c3799093b7725e6e3c9d: Status 404 returned error can't find the container with id 446e10db009b58e0517e8ac4cda761af510ff31d0322c3799093b7725e6e3c9d Oct 30 02:15:15.875907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.875866 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" event={"ID":"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0","Type":"ContainerStarted","Data":"7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e"} Oct 30 02:15:15.876397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.875944 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" event={"ID":"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0","Type":"ContainerStarted","Data":"446e10db009b58e0517e8ac4cda761af510ff31d0322c3799093b7725e6e3c9d"} Oct 30 02:15:15.876397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.876098 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:15.877683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.877661 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" event={"ID":"f4fc8538-c30f-4242-9342-ed8661810e6a","Type":"ContainerStarted","Data":"8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1"} Oct 30 02:15:15.877793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.877700 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" event={"ID":"f4fc8538-c30f-4242-9342-ed8661810e6a","Type":"ContainerStarted","Data":"db5ffda18fd6aa712b8d63c6b810a4c6a8a785090858e12580bcb49ad49f5dea"} Oct 30 02:15:15.877793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.877720 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:15.914375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.914312 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" podStartSLOduration=1.914293887 podStartE2EDuration="1.914293887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:15.91342733 +0000 UTC m=+2744.305234975" watchObservedRunningTime="2026-10-30 02:15:15.914293887 +0000 UTC m=+2744.306101522" Oct 30 02:15:15.954157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:15.954069 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" podStartSLOduration=1.9540495949999999 podStartE2EDuration="1.954049595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:15.950504736 +0000 UTC m=+2744.342312369" watchObservedRunningTime="2026-10-30 02:15:15.954049595 +0000 UTC m=+2744.345857229" Oct 30 02:15:16.699504 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:15:16.699463 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:15:18.732493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:18.732455 6275 scope.go:122] "RemoveContainer" containerID="e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4" Oct 30 02:15:19.908743 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:19.908670 6275 generic.go:356] "Generic (PLEG): container finished" podID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerID="df267bce123ad256ede45dff3ebe063d43946db52c1b605dab40c25f24af12b3" exitCode=60 Oct 30 02:15:19.909182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:19.908745 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerDied","Data":"df267bce123ad256ede45dff3ebe063d43946db52c1b605dab40c25f24af12b3"} Oct 30 02:15:19.909182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:19.908794 6275 scope.go:122] "RemoveContainer" containerID="e4f0a74019f8b3d53705f9291d8f7899a2473fb23649987ff2b5dc488ba829e4" Oct 30 02:15:19.909290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:19.909206 6275 scope.go:122] "RemoveContainer" containerID="df267bce123ad256ede45dff3ebe063d43946db52c1b605dab40c25f24af12b3" Oct 30 02:15:19.909551 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:15:19.909533 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf_odh-ai-gateway-infra(61b668be-f4da-431d-b394-a5aa8cef16c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" Oct 30 02:15:20.957391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:20.957360 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf"] Oct 30 02:15:21.122471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.122444 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:21.269064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.269028 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6\" (UniqueName: \"kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6\") pod \"61b668be-f4da-431d-b394-a5aa8cef16c3\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " Oct 30 02:15:21.269253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.269087 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca\") pod \"61b668be-f4da-431d-b394-a5aa8cef16c3\" (UID: \"61b668be-f4da-431d-b394-a5aa8cef16c3\") " Oct 30 02:15:21.269513 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.269491 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca" pod "61b668be-f4da-431d-b394-a5aa8cef16c3" (UID: "61b668be-f4da-431d-b394-a5aa8cef16c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.269728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.269701 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b668be-f4da-431d-b394-a5aa8cef16c3-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.271409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.271383 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6" pod "61b668be-f4da-431d-b394-a5aa8cef16c3" (UID: "61b668be-f4da-431d-b394-a5aa8cef16c3"). InnerVolumeSpecName "kube-api-access-5s7f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.370798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.370764 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7f6\" (UniqueName: \"kubernetes.io/projected/61b668be-f4da-431d-b394-a5aa8cef16c3-kube-api-access-5s7f6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.928074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.928038 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" Oct 30 02:15:21.928074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.928048 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf" event={"ID":"61b668be-f4da-431d-b394-a5aa8cef16c3","Type":"ContainerDied","Data":"ac03180456322d94f944e9ca2a9c6cf54728a07fff109c4715e63f4c419c276d"} Oct 30 02:15:21.928370 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.928147 6275 scope.go:122] "RemoveContainer" containerID="df267bce123ad256ede45dff3ebe063d43946db52c1b605dab40c25f24af12b3" Oct 30 02:15:21.975941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.975901 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf"] Oct 30 02:15:21.981762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:21.981736 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888775-7nwjf"] Oct 30 02:15:23.776182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:23.776148 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" path="/var/lib/kubelet/pods/61b668be-f4da-431d-b394-a5aa8cef16c3/volumes" Oct 30 02:15:23.861238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:23.861211 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:15:26.888826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:26.888797 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:15:26.889316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:26.888862 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:15:44.316195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.316157 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7"] Oct 30 02:15:44.317057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.317037 6275 cpu_manager.go:414] "containerMap: removing container" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:15:44.317153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.317058 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:15:44.317153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.317107 6275 cpu_manager.go:414] "containerMap: removing container" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:15:44.317153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.317113 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:15:44.317322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.317234 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:15:44.317322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.317250 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:15:44.322557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.322536 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.339590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.339363 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-9dfb3c-serving-cert\"" Oct 30 02:15:44.361375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.361339 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7"] Oct 30 02:15:44.418430 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.418395 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2bt\" (UniqueName: \"kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.418633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.418579 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.418633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.418611 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.418783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.418768 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.534491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.534455 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2bt\" (UniqueName: \"kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.534702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.534626 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.534702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.534660 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.534830 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.534798 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.536027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.535961 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.566486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.565397 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.566879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.566853 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.572987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.572510 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2bt\" (UniqueName: \"kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt\") pod \"maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.644190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.644154 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:44.956804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:44.956772 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7"] Oct 30 02:15:44.959179 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:15:44.959150 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236d9bfe_9a3f_4240_80a6_c01b604d31b0.slice/crio-c8b6c1b9a079b2cf1584098f230dfa020f8aea2bf1455c8d19a039b8f8598d56 WatchSource:0}: Error finding container c8b6c1b9a079b2cf1584098f230dfa020f8aea2bf1455c8d19a039b8f8598d56: Status 404 returned error can't find the container with id c8b6c1b9a079b2cf1584098f230dfa020f8aea2bf1455c8d19a039b8f8598d56 Oct 30 02:15:45.172031 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:45.171989 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" event={"ID":"236d9bfe-9a3f-4240-80a6-c01b604d31b0","Type":"ContainerStarted","Data":"c8b6c1b9a079b2cf1584098f230dfa020f8aea2bf1455c8d19a039b8f8598d56"} Oct 30 02:15:46.181210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:46.181171 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" event={"ID":"236d9bfe-9a3f-4240-80a6-c01b604d31b0","Type":"ContainerStarted","Data":"0436b5b6e40825b0dbd10e0f3b7e4295ea9840bdddaafc02559985fcd89de4b8"} Oct 30 02:15:46.181648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:46.181250 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:15:46.223525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:46.223465 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" podStartSLOduration=1.652407877 podStartE2EDuration="2.223448725s" totalImagesPullingTime="571.040848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:46.221369883 +0000 UTC m=+2774.613177513" watchObservedRunningTime="2026-10-30 02:15:46.223448725 +0000 UTC m=+2774.615256358" Oct 30 02:15:46.699531 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:15:46.699493 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:15:52.194258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:15:52.194220 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:16:11.469485 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.469439 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2"] Oct 30 02:16:11.470749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.470716 6275 cpu_manager.go:414] "containerMap: removing container" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:16:11.470749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.470747 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:16:11.470945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.470932 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b668be-f4da-431d-b394-a5aa8cef16c3" containerName="cleanup" Oct 30 02:16:11.475689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.475666 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.495153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.493508 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-9bf0e2-serving-cert\"" Oct 30 02:16:11.508442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.508409 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2"] Oct 30 02:16:11.648644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.648604 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqcw7\" (UniqueName: \"kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.648829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.648678 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.648829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.648765 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.648829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.648788 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.754553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.751995 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqcw7\" (UniqueName: \"kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.754553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.752080 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.754553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.752161 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.754553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.752191 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.754553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.753164 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.756277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.756251 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.757271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.757246 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.774447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.774414 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqcw7\" (UniqueName: \"kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7\") pod \"maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:11.794661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:11.794627 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:12.111321 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:12.111289 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2"] Oct 30 02:16:12.114153 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:16:12.114107 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb96295_da6f_41c5_9316_9ed4492c20c1.slice/crio-63e769d0153bf71a06c50c21bff8bdbb084709be971c5fde1b7a7ce6acc0f738 WatchSource:0}: Error finding container 63e769d0153bf71a06c50c21bff8bdbb084709be971c5fde1b7a7ce6acc0f738: Status 404 returned error can't find the container with id 63e769d0153bf71a06c50c21bff8bdbb084709be971c5fde1b7a7ce6acc0f738 Oct 30 02:16:12.451168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:12.451114 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" event={"ID":"1fb96295-da6f-41c5-9316-9ed4492c20c1","Type":"ContainerStarted","Data":"63e769d0153bf71a06c50c21bff8bdbb084709be971c5fde1b7a7ce6acc0f738"} Oct 30 02:16:13.480231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:13.479616 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" event={"ID":"1fb96295-da6f-41c5-9316-9ed4492c20c1","Type":"ContainerStarted","Data":"cc39aa585b5b0e852828077afdc20e065edd67c3fa924c934e0362a1c40170da"} Oct 30 02:16:13.480231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:13.479685 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:13.524414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:13.524356 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" podStartSLOduration=2.210399837 podStartE2EDuration="2.524332561s" totalImagesPullingTime="313.932724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:13.524275715 +0000 UTC m=+2801.916083350" watchObservedRunningTime="2026-10-30 02:16:13.524332561 +0000 UTC m=+2801.916140191" Oct 30 02:16:14.488146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:14.488014 6275 generic.go:356] "Generic (PLEG): container finished" podID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerID="49c562bea926f8a407dc2a17b91ebf66400f7d0259fee0ea4dd2fa8074a08dfa" exitCode=0 Oct 30 02:16:14.488146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:14.488089 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" event={"ID":"5c06325f-3df6-48c3-8f02-53aeef8a2de2","Type":"ContainerDied","Data":"49c562bea926f8a407dc2a17b91ebf66400f7d0259fee0ea4dd2fa8074a08dfa"} Oct 30 02:16:14.488876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:14.488847 6275 scope.go:122] "RemoveContainer" containerID="49c562bea926f8a407dc2a17b91ebf66400f7d0259fee0ea4dd2fa8074a08dfa" Oct 30 02:16:14.490739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:14.490705 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:16:14.490822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:14.490800 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:16:14.490878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:14.490849 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:16:15.497445 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:15.497403 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" event={"ID":"5c06325f-3df6-48c3-8f02-53aeef8a2de2","Type":"ContainerStarted","Data":"bcc7e7884a17834a910be4a279d396457c4ce6ed8a3b2b6229848380c1ce0613"} Oct 30 02:16:16.018523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:16.018493 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:16:16.018720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:16.018660 6275 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:16:16.018720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:16.018703 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:16:16.731755 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:16:16.731720 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:16:17.018194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:17.018164 6275 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:16:17.018386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:17.018220 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:16:18.019048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:18.019015 6275 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:16:18.019468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:18.019077 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:16:19.018641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:19.018607 6275 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:16:19.018876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:19.018659 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:16:19.494070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:19.493986 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:16:20.018802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:20.018764 6275 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:16:20.019024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:20.018824 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:16:21.051176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:21.051116 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:16:21.052315 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:21.052288 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:16:21.053365 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:16:21.053344 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:16:46.745442 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:16:46.745372 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:17:16.697197 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:17:16.697156 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:17:46.698739 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:17:46.698660 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:18:14.408427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.408393 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2"] Oct 30 02:18:14.409063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.408721 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" podUID="1fb96295-da6f-41c5-9316-9ed4492c20c1" containerName="maas-api" containerID="cri-o://cc39aa585b5b0e852828077afdc20e065edd67c3fa924c934e0362a1c40170da" gracePeriod=30 Oct 30 02:18:14.488775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.488730 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" podUID="1fb96295-da6f-41c5-9316-9ed4492c20c1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.218:8443/health\": dial tcp 10.134.0.218:8443: connect: connection refused" Oct 30 02:18:14.538403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.538360 6275 generic.go:356] "Generic (PLEG): container finished" podID="1fb96295-da6f-41c5-9316-9ed4492c20c1" containerID="cc39aa585b5b0e852828077afdc20e065edd67c3fa924c934e0362a1c40170da" exitCode=0 Oct 30 02:18:14.538551 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.538415 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" event={"ID":"1fb96295-da6f-41c5-9316-9ed4492c20c1","Type":"ContainerDied","Data":"cc39aa585b5b0e852828077afdc20e065edd67c3fa924c934e0362a1c40170da"} Oct 30 02:18:14.751741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.751716 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:18:14.928659 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.928614 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7\" (UniqueName: \"kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7\") pod \"1fb96295-da6f-41c5-9316-9ed4492c20c1\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " Oct 30 02:18:14.928867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.928703 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs\") pod \"1fb96295-da6f-41c5-9316-9ed4492c20c1\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " Oct 30 02:18:14.928867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.928740 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls\") pod \"1fb96295-da6f-41c5-9316-9ed4492c20c1\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " Oct 30 02:18:14.928987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.928945 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca\") pod \"1fb96295-da6f-41c5-9316-9ed4492c20c1\" (UID: \"1fb96295-da6f-41c5-9316-9ed4492c20c1\") " Oct 30 02:18:14.933806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.933705 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7" pod "1fb96295-da6f-41c5-9316-9ed4492c20c1" (UID: "1fb96295-da6f-41c5-9316-9ed4492c20c1"). InnerVolumeSpecName "kube-api-access-bqcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:14.937909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.937611 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs" pod "1fb96295-da6f-41c5-9316-9ed4492c20c1" (UID: "1fb96295-da6f-41c5-9316-9ed4492c20c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:14.943698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.943661 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls" pod "1fb96295-da6f-41c5-9316-9ed4492c20c1" (UID: "1fb96295-da6f-41c5-9316-9ed4492c20c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:14.944314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:14.944269 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca" pod "1fb96295-da6f-41c5-9316-9ed4492c20c1" (UID: "1fb96295-da6f-41c5-9316-9ed4492c20c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:15.031049 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.031012 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb96295-da6f-41c5-9316-9ed4492c20c1-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:18:15.031049 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.031046 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqcw7\" (UniqueName: \"kubernetes.io/projected/1fb96295-da6f-41c5-9316-9ed4492c20c1-kube-api-access-bqcw7\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:18:15.031297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.031061 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:18:15.031297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.031074 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb96295-da6f-41c5-9316-9ed4492c20c1-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:18:15.547220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.547189 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" Oct 30 02:18:15.547220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.547190 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2" event={"ID":"1fb96295-da6f-41c5-9316-9ed4492c20c1","Type":"ContainerDied","Data":"63e769d0153bf71a06c50c21bff8bdbb084709be971c5fde1b7a7ce6acc0f738"} Oct 30 02:18:15.547716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.547266 6275 scope.go:122] "RemoveContainer" containerID="cc39aa585b5b0e852828077afdc20e065edd67c3fa924c934e0362a1c40170da" Oct 30 02:18:15.594457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.594424 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2"] Oct 30 02:18:15.604109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.604079 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9bf0e2-ff589f586-4k5p2"] Oct 30 02:18:15.740734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:15.740694 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb96295-da6f-41c5-9316-9ed4492c20c1" path="/var/lib/kubelet/pods/1fb96295-da6f-41c5-9316-9ed4492c20c1/volumes" Oct 30 02:18:16.700069 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:18:16.700032 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:18:17.653951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.653909 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv"] Oct 30 02:18:17.654917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.654891 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1fb96295-da6f-41c5-9316-9ed4492c20c1" containerName="maas-api" Oct 30 02:18:17.654917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.654914 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb96295-da6f-41c5-9316-9ed4492c20c1" containerName="maas-api" Oct 30 02:18:17.655161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.655135 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb96295-da6f-41c5-9316-9ed4492c20c1" containerName="maas-api" Oct 30 02:18:17.660328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.660306 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.675190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.675091 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-1991e0bfc294\"" Oct 30 02:18:17.675190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.675099 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-d2862aa4fcc5\"" Oct 30 02:18:17.675190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.675157 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-9dfb3c\"" Oct 30 02:18:17.675429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.675227 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"routing-overlay\"" Oct 30 02:18:17.685959 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.685930 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"openshift-service-ca.crt\"" Oct 30 02:18:17.685959 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.685940 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"kube-root-ca.crt\"" Oct 30 02:18:17.686220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.685946 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-437db4025c2d\"" Oct 30 02:18:17.686220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.686100 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"payload-processing-external-model-e2e--1ed6558d-dockercfg-xn4z5\"" Oct 30 02:18:17.692448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.692421 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv"] Oct 30 02:18:17.783368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.783335 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.783814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.783472 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.783814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.783519 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjlw\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.783814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.783558 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.884674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.884640 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.884874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.884724 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.884874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.884749 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjlw\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.884874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.884774 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.885332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.885305 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.885534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.885511 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.888037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.888006 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.907574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.907489 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjlw\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:17.974918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:17.974881 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:18.285434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:18.285412 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv"] Oct 30 02:18:18.288089 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:18:18.288058 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8603fa86_fbff_4a00_849d_a2e57790630f.slice/crio-812d0357972763d24ff24de6d4f82dbe56b3ed23e8a15c74e17ce1402b85e963 WatchSource:0}: Error finding container 812d0357972763d24ff24de6d4f82dbe56b3ed23e8a15c74e17ce1402b85e963: Status 404 returned error can't find the container with id 812d0357972763d24ff24de6d4f82dbe56b3ed23e8a15c74e17ce1402b85e963 Oct 30 02:18:18.576358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:18.576266 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" event={"ID":"8603fa86-fbff-4a00-849d-a2e57790630f","Type":"ContainerStarted","Data":"1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e"} Oct 30 02:18:18.576358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:18.576319 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" event={"ID":"8603fa86-fbff-4a00-849d-a2e57790630f","Type":"ContainerStarted","Data":"812d0357972763d24ff24de6d4f82dbe56b3ed23e8a15c74e17ce1402b85e963"} Oct 30 02:18:18.576358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:18.576339 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:18.616162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:18.616095 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" podStartSLOduration=1.6160823309999999 podStartE2EDuration="1.616082331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:18.613603003 +0000 UTC m=+2927.005410636" watchObservedRunningTime="2026-10-30 02:18:18.616082331 +0000 UTC m=+2927.007889964" Oct 30 02:18:29.531069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.531034 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7"] Oct 30 02:18:29.535860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.535843 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.563650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.563599 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-95f40f\"" Oct 30 02:18:29.563650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.563609 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-95f40f-dockercfg-rtrjt\"" Oct 30 02:18:29.568828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.568796 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7"] Oct 30 02:18:29.588311 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.588280 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:18:29.725276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.696622 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrcs\" (UniqueName: \"kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs\") pod \"payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.727061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.726931 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.728629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.728556 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2"] Oct 30 02:18:29.736836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.735926 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:29.772511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.772475 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2"] Oct 30 02:18:29.828708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.828628 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:29.828708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.828694 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.828901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.828778 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrcs\" (UniqueName: \"kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs\") pod \"payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.828901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.828872 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7d8n\" (UniqueName: \"kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n\") pod \"payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:29.829608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.829588 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.851107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.851073 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrcs\" (UniqueName: \"kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs\") pod \"payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:29.930403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.930370 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7d8n\" (UniqueName: \"kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n\") pod \"payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:29.930605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.930446 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:29.931286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.931265 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:29.951972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:29.951947 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7d8n\" (UniqueName: \"kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n\") pod \"payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:30.073387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.073338 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:30.151676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.151641 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:30.387347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.387321 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2"] Oct 30 02:18:30.389800 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:18:30.389772 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bdad21_9f15_43a1_8af7_1e22cf49aaa9.slice/crio-e910ebdb65ab1597f1a521d8b7c6e81bc5ef87c4a8d8c1835ec474863267584a WatchSource:0}: Error finding container e910ebdb65ab1597f1a521d8b7c6e81bc5ef87c4a8d8c1835ec474863267584a: Status 404 returned error can't find the container with id e910ebdb65ab1597f1a521d8b7c6e81bc5ef87c4a8d8c1835ec474863267584a Oct 30 02:18:30.476649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.476615 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7"] Oct 30 02:18:30.480473 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:18:30.480442 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d23e69_56ba_4550_89c9_d66cc6a895ed.slice/crio-811d62955a140225ddfa2ace9f99e384cef4dedfa89d046c521800fb4b1dca59 WatchSource:0}: Error finding container 811d62955a140225ddfa2ace9f99e384cef4dedfa89d046c521800fb4b1dca59: Status 404 returned error can't find the container with id 811d62955a140225ddfa2ace9f99e384cef4dedfa89d046c521800fb4b1dca59 Oct 30 02:18:30.717387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.717289 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" event={"ID":"40d23e69-56ba-4550-89c9-d66cc6a895ed","Type":"ContainerStarted","Data":"570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a"} Oct 30 02:18:30.717387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.717358 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:30.717387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.717382 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" event={"ID":"40d23e69-56ba-4550-89c9-d66cc6a895ed","Type":"ContainerStarted","Data":"811d62955a140225ddfa2ace9f99e384cef4dedfa89d046c521800fb4b1dca59"} Oct 30 02:18:30.719011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.718985 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" event={"ID":"01bdad21-9f15-43a1-8af7-1e22cf49aaa9","Type":"ContainerStarted","Data":"d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4"} Oct 30 02:18:30.719157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.719025 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" event={"ID":"01bdad21-9f15-43a1-8af7-1e22cf49aaa9","Type":"ContainerStarted","Data":"e910ebdb65ab1597f1a521d8b7c6e81bc5ef87c4a8d8c1835ec474863267584a"} Oct 30 02:18:30.719157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.719055 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:30.777728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.777671 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" podStartSLOduration=1.777655803 podStartE2EDuration="1.777655803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:30.774156203 +0000 UTC m=+2939.165963833" watchObservedRunningTime="2026-10-30 02:18:30.777655803 +0000 UTC m=+2939.169463474" Oct 30 02:18:30.816714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:30.816592 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" podStartSLOduration=1.8165749070000001 podStartE2EDuration="1.816574907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:30.813524234 +0000 UTC m=+2939.205331866" watchObservedRunningTime="2026-10-30 02:18:30.816574907 +0000 UTC m=+2939.208382541" Oct 30 02:18:34.923060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:34.923027 6275 scope.go:122] "RemoveContainer" containerID="bd2346070aa12b9e0254c51780e34b3bab8de242d1d6a3161d57157232136598" Oct 30 02:18:38.070999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.070954 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr"] Oct 30 02:18:38.077102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.077078 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.090265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.090236 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6b8bfa-openshift-default-dockercfg-gc4jw\"" Oct 30 02:18:38.104975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.104939 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr"] Oct 30 02:18:38.163718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.163677 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.163874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.163805 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.163961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.163869 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.163961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.163903 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.164054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.164023 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.164173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.164149 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xdm\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.164273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.164248 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.164317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.164293 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.164412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.164397 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265345 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265440 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265480 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265601 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265648 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265728 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265805 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xdm\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265844 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.265896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265884 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.266006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265921 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.266006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265930 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.266006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.265943 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.266230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.266209 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.266277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.266245 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.268244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.268221 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.268388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.268368 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.287672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.287636 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.287792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.287761 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xdm\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.391980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.391885 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:38.718665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.718633 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr"] Oct 30 02:18:38.720723 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:18:38.720699 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16341446_568b_4d49_be87_c4be1c3bdc46.slice/crio-0ff6c4ad27c411f9e7a90eccced8fd4abef354308fc738de3c13d0f10dfb0efc WatchSource:0}: Error finding container 0ff6c4ad27c411f9e7a90eccced8fd4abef354308fc738de3c13d0f10dfb0efc: Status 404 returned error can't find the container with id 0ff6c4ad27c411f9e7a90eccced8fd4abef354308fc738de3c13d0f10dfb0efc Oct 30 02:18:38.723545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.723511 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:18:38.723655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.723579 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:18:38.723655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.723609 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:18:38.812472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.812435 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" event={"ID":"16341446-568b-4d49-be87-c4be1c3bdc46","Type":"ContainerStarted","Data":"72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9"} Oct 30 02:18:38.812657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.812490 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" event={"ID":"16341446-568b-4d49-be87-c4be1c3bdc46","Type":"ContainerStarted","Data":"0ff6c4ad27c411f9e7a90eccced8fd4abef354308fc738de3c13d0f10dfb0efc"} Oct 30 02:18:38.859529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:38.859458 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podStartSLOduration=0.85942805 podStartE2EDuration="859.42805ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:38.855163534 +0000 UTC m=+2947.246971164" watchObservedRunningTime="2026-10-30 02:18:38.85942805 +0000 UTC m=+2947.251235683" Oct 30 02:18:39.392582 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:39.392529 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:39.394249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:39.394216 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:18:39.394409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:39.394272 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 02:18:40.392664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:40.392625 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:18:40.393070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:40.392683 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 02:18:41.392533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:41.392501 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:18:41.392723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:41.392553 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 02:18:41.729215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:41.729102 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:18:41.729215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:41.729206 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:18:42.392475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:42.392434 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:18:42.392659 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:42.392501 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 02:18:43.392420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:43.392384 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:18:43.392831 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:43.392454 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 02:18:44.392234 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:44.392197 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:18:44.392413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:44.392249 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 02:18:45.398502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:45.398467 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:45.398916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:45.398884 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:45.399534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:18:45.399513 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:18:46.697733 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:18:46.697697 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:19:16.698877 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:19:16.698790 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:19:32.397011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:19:32.396981 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:19:32.403335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:19:32.403310 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:19:46.698318 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:19:46.698274 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:20:06.097545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:06.097505 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv"] Oct 30 02:20:11.098588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.098534 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" podUID="8603fa86-fbff-4a00-849d-a2e57790630f" containerName="payload-processing" containerID="cri-o://1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e" gracePeriod=25 Oct 30 02:20:11.403420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.403389 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:20:11.508703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.508671 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume\") pod \"8603fa86-fbff-4a00-849d-a2e57790630f\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " Oct 30 02:20:11.508907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.508726 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume\") pod \"8603fa86-fbff-4a00-849d-a2e57790630f\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " Oct 30 02:20:11.508907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.508776 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials\") pod \"8603fa86-fbff-4a00-849d-a2e57790630f\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " Oct 30 02:20:11.508907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.508863 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw\") pod \"8603fa86-fbff-4a00-849d-a2e57790630f\" (UID: \"8603fa86-fbff-4a00-849d-a2e57790630f\") " Oct 30 02:20:11.509117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.509094 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume" pod "8603fa86-fbff-4a00-849d-a2e57790630f" (UID: "8603fa86-fbff-4a00-849d-a2e57790630f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:11.509117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.509102 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume" pod "8603fa86-fbff-4a00-849d-a2e57790630f" (UID: "8603fa86-fbff-4a00-849d-a2e57790630f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:11.509548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.509529 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.509614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.509552 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8603fa86-fbff-4a00-849d-a2e57790630f-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.511293 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.511271 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials" pod "8603fa86-fbff-4a00-849d-a2e57790630f" (UID: "8603fa86-fbff-4a00-849d-a2e57790630f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.511375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.511334 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw" pod "8603fa86-fbff-4a00-849d-a2e57790630f" (UID: "8603fa86-fbff-4a00-849d-a2e57790630f"). InnerVolumeSpecName "kube-api-access-mcjlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.615206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.613552 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.615206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.613595 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjlw\" (UniqueName: \"kubernetes.io/projected/8603fa86-fbff-4a00-849d-a2e57790630f-kube-api-access-mcjlw\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.770297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.770264 6275 generic.go:356] "Generic (PLEG): container finished" podID="8603fa86-fbff-4a00-849d-a2e57790630f" containerID="1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e" exitCode=0 Oct 30 02:20:11.770474 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.770359 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" Oct 30 02:20:11.771070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.771048 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" event={"ID":"8603fa86-fbff-4a00-849d-a2e57790630f","Type":"ContainerDied","Data":"1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e"} Oct 30 02:20:11.771279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.771263 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv" event={"ID":"8603fa86-fbff-4a00-849d-a2e57790630f","Type":"ContainerDied","Data":"812d0357972763d24ff24de6d4f82dbe56b3ed23e8a15c74e17ce1402b85e963"} Oct 30 02:20:11.771435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.771395 6275 scope.go:122] "RemoveContainer" containerID="1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e" Oct 30 02:20:11.784947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.784928 6275 scope.go:122] "RemoveContainer" containerID="1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e" Oct 30 02:20:11.786929 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:11.786905 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e\": container with ID starting with 1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e not found: ID does not exist" containerID="1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e" Oct 30 02:20:11.787026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.786938 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e"} err="failed to get container status \"1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e\": rpc error: code = NotFound desc = could not find container \"1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e\": container with ID starting with 1665add03357440dbccd92e6365388965d5a50b95b3a9213013f83be9d9cba9e not found: ID does not exist" Oct 30 02:20:11.822060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.822016 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv"] Oct 30 02:20:11.827988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:11.827958 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5dc7prqmv"] Oct 30 02:20:12.950958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.950919 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-565954b4c8-xdflc"] Oct 30 02:20:12.951721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.951702 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8603fa86-fbff-4a00-849d-a2e57790630f" containerName="payload-processing" Oct 30 02:20:12.951721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.951720 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8603fa86-fbff-4a00-849d-a2e57790630f" containerName="payload-processing" Oct 30 02:20:12.951859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.951831 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8603fa86-fbff-4a00-849d-a2e57790630f" containerName="payload-processing" Oct 30 02:20:12.957020 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.957000 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:12.971635 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.971611 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:20:12.982320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.982291 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:20:12.982320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.982291 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:20:12.984731 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.984707 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d99f27852cee\"" Oct 30 02:20:12.984866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.984717 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:20:12.984866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.984850 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gf5z7\"" Oct 30 02:20:12.988976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:12.988952 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-565954b4c8-xdflc"] Oct 30 02:20:13.116792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.116750 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.138145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.117205 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.138520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.138488 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.138827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.138806 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9dk\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.241199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.240966 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.241199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.241107 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.241445 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.241201 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.241445 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.241241 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9dk\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.243091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.243060 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.243324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.243185 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.244679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.244636 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.263903 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.263860 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9dk\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk\") pod \"payload-processing-external-model-565954b4c8-xdflc\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.271527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.271501 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.589085 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.589061 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-565954b4c8-xdflc"] Oct 30 02:20:13.590443 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:20:13.590421 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaa804a_b9c1_4b97_8f91_b64f3c75645d.slice/crio-2c47b52d7d22c65bd15af4935d3733ca11067ba420f78b179734178c4b481ee1 WatchSource:0}: Error finding container 2c47b52d7d22c65bd15af4935d3733ca11067ba420f78b179734178c4b481ee1: Status 404 returned error can't find the container with id 2c47b52d7d22c65bd15af4935d3733ca11067ba420f78b179734178c4b481ee1 Oct 30 02:20:13.741881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.741843 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8603fa86-fbff-4a00-849d-a2e57790630f" path="/var/lib/kubelet/pods/8603fa86-fbff-4a00-849d-a2e57790630f/volumes" Oct 30 02:20:13.787827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.787789 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" event={"ID":"4aaa804a-b9c1-4b97-8f91-b64f3c75645d","Type":"ContainerStarted","Data":"21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d"} Oct 30 02:20:13.788019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.787857 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" event={"ID":"4aaa804a-b9c1-4b97-8f91-b64f3c75645d","Type":"ContainerStarted","Data":"2c47b52d7d22c65bd15af4935d3733ca11067ba420f78b179734178c4b481ee1"} Oct 30 02:20:13.788019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.787990 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:13.831978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:13.831848 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" podStartSLOduration=1.8318312140000002 podStartE2EDuration="1.831831214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:13.829026629 +0000 UTC m=+3042.220834263" watchObservedRunningTime="2026-10-30 02:20:13.831831214 +0000 UTC m=+3042.223638852" Oct 30 02:20:16.698502 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:16.698465 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:20:24.800117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:24.800085 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:20:25.373751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.373706 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9"] Oct 30 02:20:25.379961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.379930 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.394886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.394857 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-e2e--b7f31742-dockercfg-cjhl5\"" Oct 30 02:20:25.396063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.396039 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"e2e-cfl-544dc5-provider-cred\"" Oct 30 02:20:25.399164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.399117 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9"] Oct 30 02:20:25.399855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.399834 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"openshift-service-ca.crt\"" Oct 30 02:20:25.399981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.399958 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"routing-overlay\"" Oct 30 02:20:25.433183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.425645 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"kube-root-ca.crt\"" Oct 30 02:20:25.433183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.425791 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6b8bfa\"" Oct 30 02:20:25.583549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.583510 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"agc-cred-2a752186c169\"" Oct 30 02:20:25.692587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.692499 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.692587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.692564 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.692850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.692693 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.692850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.692768 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdk2r\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.797600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.797554 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.798877 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.798737 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.799146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.799112 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdk2r\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.799553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.799533 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.799704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.799687 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.799896 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:25.799880 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:25.800013 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:25.800001 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9: secret "agc-cred-2a752186c169" not found Oct 30 02:20:25.800220 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:25.800206 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials podName:9815f630-e43c-4c46-8c08-4741bb52074d nodeName:}" failed. No retries permitted until 2026-10-30 02:20:26.300174671 +0000 UTC m=+3054.691982301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:25.815374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.814362 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:25.855509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:25.855469 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdk2r\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:26.325176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:26.325101 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:26.325374 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:26.325243 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:26.325374 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:26.325274 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9: secret "agc-cred-2a752186c169" not found Oct 30 02:20:26.325374 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:26.325360 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials podName:9815f630-e43c-4c46-8c08-4741bb52074d nodeName:}" failed. No retries permitted until 2026-10-30 02:20:27.325339518 +0000 UTC m=+3055.717147146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.361574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.361542 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:27.361966 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:27.361671 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.361966 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:27.361704 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9: secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.361966 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:27.361790 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials podName:9815f630-e43c-4c46-8c08-4741bb52074d nodeName:}" failed. No retries permitted until 2026-10-30 02:20:29.361769197 +0000 UTC m=+3057.753576831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.399735 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.399695 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9"] Oct 30 02:20:27.406422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.406396 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.430484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.430449 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9"] Oct 30 02:20:27.462653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.462613 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7z5z\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.462861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.462785 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.462925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.462905 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.462976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.462954 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.573173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.570163 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7z5z\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.573173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.570337 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.573173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.570446 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.573173 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.570514 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.588143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.577292 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.588143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.583291 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.600893 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.600807 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.606985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.606949 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7z5z\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:27.722942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:27.722833 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:28.049950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:28.049914 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9"] Oct 30 02:20:28.053364 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:20:28.053331 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ec5129_19be_4b66_a957_10cc7825e237.slice/crio-14db95e623378722ea68c48caac9776b25d76f2d8febd64cec96afadaff70b83 WatchSource:0}: Error finding container 14db95e623378722ea68c48caac9776b25d76f2d8febd64cec96afadaff70b83: Status 404 returned error can't find the container with id 14db95e623378722ea68c48caac9776b25d76f2d8febd64cec96afadaff70b83 Oct 30 02:20:28.968961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:28.968917 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" event={"ID":"45ec5129-19be-4b66-a957-10cc7825e237","Type":"ContainerStarted","Data":"1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470"} Oct 30 02:20:28.969549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:28.969522 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:28.969660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:28.969552 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" event={"ID":"45ec5129-19be-4b66-a957-10cc7825e237","Type":"ContainerStarted","Data":"14db95e623378722ea68c48caac9776b25d76f2d8febd64cec96afadaff70b83"} Oct 30 02:20:29.013728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:29.013673 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" podStartSLOduration=2.013659025 podStartE2EDuration="2.013659025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:29.012198301 +0000 UTC m=+3057.404005935" watchObservedRunningTime="2026-10-30 02:20:29.013659025 +0000 UTC m=+3057.405466657" Oct 30 02:20:29.423378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:29.423342 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:29.423586 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:29.423515 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:29.423586 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:29.423540 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9: secret "agc-cred-2a752186c169" not found Oct 30 02:20:29.423714 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:29.423606 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials podName:9815f630-e43c-4c46-8c08-4741bb52074d nodeName:}" failed. No retries permitted until 2026-10-30 02:20:33.423585803 +0000 UTC m=+3061.815393432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:33.503541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:33.503498 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:33.503961 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:33.503643 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:33.503961 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:33.503668 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9: secret "agc-cred-2a752186c169" not found Oct 30 02:20:33.503961 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:33.503735 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials podName:9815f630-e43c-4c46-8c08-4741bb52074d nodeName:}" failed. No retries permitted until 2026-10-30 02:20:41.503720934 +0000 UTC m=+3069.895528545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:39.980042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:39.980011 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:20:40.054333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:40.054295 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9"] Oct 30 02:20:40.054824 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:40.054797 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" podUID="9815f630-e43c-4c46-8c08-4741bb52074d" Oct 30 02:20:41.064796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.064764 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:41.074130 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.074086 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:41.216797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.216759 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r\") pod \"9815f630-e43c-4c46-8c08-4741bb52074d\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " Oct 30 02:20:41.216976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.216806 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume\") pod \"9815f630-e43c-4c46-8c08-4741bb52074d\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " Oct 30 02:20:41.216976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.216969 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume\") pod \"9815f630-e43c-4c46-8c08-4741bb52074d\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " Oct 30 02:20:41.217366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.217340 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume" pod "9815f630-e43c-4c46-8c08-4741bb52074d" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:41.217466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.217399 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume" pod "9815f630-e43c-4c46-8c08-4741bb52074d" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:41.217732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.217715 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:41.217778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.217735 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9815f630-e43c-4c46-8c08-4741bb52074d-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:41.219059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.219037 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r" pod "9815f630-e43c-4c46-8c08-4741bb52074d" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d"). InnerVolumeSpecName "kube-api-access-bdk2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:41.319091 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.319005 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdk2r\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-kube-api-access-bdk2r\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:41.521175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:41.521114 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9\" (UID: \"9815f630-e43c-4c46-8c08-4741bb52074d\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:41.521367 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:41.521266 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:41.521367 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:41.521291 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9: secret "agc-cred-2a752186c169" not found Oct 30 02:20:41.521367 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:41.521357 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials podName:9815f630-e43c-4c46-8c08-4741bb52074d nodeName:}" failed. No retries permitted until 2026-10-30 02:20:57.521341512 +0000 UTC m=+3085.913149126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" (UID: "9815f630-e43c-4c46-8c08-4741bb52074d") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:42.072913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:42.072881 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9" Oct 30 02:20:42.151094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:42.151056 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9"] Oct 30 02:20:42.155691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:42.155658 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-65786l4f9"] Oct 30 02:20:42.231257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:42.231220 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9815f630-e43c-4c46-8c08-4741bb52074d-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:20:43.750777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:20:43.750741 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9815f630-e43c-4c46-8c08-4741bb52074d" path="/var/lib/kubelet/pods/9815f630-e43c-4c46-8c08-4741bb52074d/volumes" Oct 30 02:20:46.698323 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:20:46.698290 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:21:00.954583 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:00.954539 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd"] Oct 30 02:21:00.967478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:00.967448 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.038469 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd"] Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.046692 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.046787 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.046841 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.046873 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.046900 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r564j\" (UniqueName: \"kubernetes.io/projected/173adfa0-45d6-4a44-b36d-f47bd184592a-kube-api-access-r564j\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.047040 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.047163 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173adfa0-45d6-4a44-b36d-f47bd184592a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.047534 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.051148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.047576 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.095827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.095787 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t"] Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.149808 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.149853 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.149883 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.149958 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.150000 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.150033 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.150062 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r564j\" (UniqueName: \"kubernetes.io/projected/173adfa0-45d6-4a44-b36d-f47bd184592a-kube-api-access-r564j\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.150218 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.150322 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173adfa0-45d6-4a44-b36d-f47bd184592a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.151144 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173adfa0-45d6-4a44-b36d-f47bd184592a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.151441 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.151661 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.163143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.151883 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.184448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.184412 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.210314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.210245 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265314 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265441 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265468 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265519 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265572 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265622 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ppb\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265679 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265845 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.294286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.265883 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.393539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.393495 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.422141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.393824 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.422467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.422420 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.422675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.422655 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.422841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.422818 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.423014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.422995 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ppb\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.423256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.423228 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.423694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.423669 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.536496 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.434068 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.434538 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.435152 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.459497 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.536144 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.554152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.434326 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.563009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.558322 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ppb\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.563009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.558893 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.563009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.559351 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.563009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.559852 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173adfa0-45d6-4a44-b36d-f47bd184592a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.563009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.561309 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.563009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.561731 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-rgk4t\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.568572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.568526 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r564j\" (UniqueName: \"kubernetes.io/projected/173adfa0-45d6-4a44-b36d-f47bd184592a-kube-api-access-r564j\") pod \"maas-default-gateway-openshift-default-75d7c785f-72kzd\" (UID: \"173adfa0-45d6-4a44-b36d-f47bd184592a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.572980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.572958 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t"] Oct 30 02:21:01.590784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.590284 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:01.844976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.844888 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:01.951695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.950224 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd"] Oct 30 02:21:01.953197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.953036 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:21:01.953197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.953113 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:21:01.953197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:01.953169 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:21:02.256225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.256165 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t"] Oct 30 02:21:02.259726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.259675 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:21:02.259867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.259783 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:21:02.259867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.259829 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:21:02.552730 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.552690 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" event={"ID":"f709c917-3b09-401e-a435-c5229dc8331a","Type":"ContainerStarted","Data":"bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc"} Oct 30 02:21:02.552928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.552752 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" event={"ID":"f709c917-3b09-401e-a435-c5229dc8331a","Type":"ContainerStarted","Data":"9873973690a2ecf124050742bc85dbe2bf62745d1ad93f62b99b48f0aecdf4c1"} Oct 30 02:21:02.555610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.555582 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" event={"ID":"173adfa0-45d6-4a44-b36d-f47bd184592a","Type":"ContainerStarted","Data":"3acf15289d42eb47d50bc2af441fb63bfe098bd4d4d3cb364fae993e3a4f0c6e"} Oct 30 02:21:02.555748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.555626 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" event={"ID":"173adfa0-45d6-4a44-b36d-f47bd184592a","Type":"ContainerStarted","Data":"92fb0a415343b4df65e6a4f5077bd116c4ea365c8cbfea2c682d793088eec035"} Oct 30 02:21:02.591759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.591701 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:02.597646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.597446 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podStartSLOduration=2.597427893 podStartE2EDuration="2.597427893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:02.594802189 +0000 UTC m=+3090.986609823" watchObservedRunningTime="2026-10-30 02:21:02.597427893 +0000 UTC m=+3090.989235527" Oct 30 02:21:02.643203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.643111 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podStartSLOduration=2.643090107 podStartE2EDuration="2.643090107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:02.639838964 +0000 UTC m=+3091.031646610" watchObservedRunningTime="2026-10-30 02:21:02.643090107 +0000 UTC m=+3091.034897740" Oct 30 02:21:02.845287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:02.845202 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:03.614145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:03.606739 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:03.614145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:03.606802 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:03.853458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:03.853412 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:03.853722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:03.853688 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:04.592182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:04.591780 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:04.592182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:04.591853 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:04.859317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:04.852328 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:04.859317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:04.852393 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:05.593144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:05.591599 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:05.593144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:05.591658 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:05.863204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:05.862210 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:05.863204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:05.862270 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:06.591557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:06.591517 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:06.591727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:06.591579 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:06.851205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:06.848523 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:06.851205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:06.848589 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:07.601149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:07.597099 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:07.601149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:07.597199 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:07.861206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:07.846560 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:07.861206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:07.846641 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:08.591087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:08.591045 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:08.591347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:08.591101 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:08.845335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:08.845238 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:08.845335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:08.845300 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:09.591811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:09.591776 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-72kzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:09.592032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:09.591826 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" podUID="173adfa0-45d6-4a44-b36d-f47bd184592a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 02:21:09.846013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:09.845921 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rgk4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 02:21:09.846013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:09.845990 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 02:21:11.158867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:11.158827 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:11.159482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:11.159459 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:11.160506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:11.160480 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-72kzd" Oct 30 02:21:11.182440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:11.182406 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:11.182790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:11.182772 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:11.185874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:11.185851 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:21:16.423165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:16.423111 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-565954b4c8-xdflc"] Oct 30 02:21:16.697854 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:21:16.697774 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:21:20.329749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.329708 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz"] Oct 30 02:21:20.334821 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.334795 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.350498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.350473 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hhnxh\"" Oct 30 02:21:20.365889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.365862 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz"] Oct 30 02:21:20.376479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.376454 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.376621 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.376499 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.376755 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.376731 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxskj\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.376918 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.376899 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.449411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.449370 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2afd06b10b99\"" Oct 30 02:21:20.478519 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.478483 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxskj\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.478701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.478626 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.478788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.478772 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.478849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.478806 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.479535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.479506 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.479683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.479510 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.481458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.481435 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.507607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.507578 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxskj\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj\") pod \"payload-processing-external-model-5db9b8ddc4-psvgz\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:20.660195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:20.659418 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:21.078865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.078840 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz"] Oct 30 02:21:21.081774 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:21:21.081747 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd984fd98_a76a_4221_b04e_26ec74dd352a.slice/crio-e34bcb019ec09cda1794a653ffed580f6753dc08b6029cb0f59a6fdb328c8aac WatchSource:0}: Error finding container e34bcb019ec09cda1794a653ffed580f6753dc08b6029cb0f59a6fdb328c8aac: Status 404 returned error can't find the container with id e34bcb019ec09cda1794a653ffed580f6753dc08b6029cb0f59a6fdb328c8aac Oct 30 02:21:21.424227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.424138 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" podUID="4aaa804a-b9c1-4b97-8f91-b64f3c75645d" containerName="payload-processing" containerID="cri-o://21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d" gracePeriod=25 Oct 30 02:21:21.694769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.694706 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:21:21.735028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735002 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume\") pod \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " Oct 30 02:21:21.735209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735159 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials\") pod \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " Oct 30 02:21:21.735209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735178 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume\") pod \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " Oct 30 02:21:21.735342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735253 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk\") pod \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\" (UID: \"4aaa804a-b9c1-4b97-8f91-b64f3c75645d\") " Oct 30 02:21:21.735967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735440 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume" pod "4aaa804a-b9c1-4b97-8f91-b64f3c75645d" (UID: "4aaa804a-b9c1-4b97-8f91-b64f3c75645d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:21.735967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735617 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume" pod "4aaa804a-b9c1-4b97-8f91-b64f3c75645d" (UID: "4aaa804a-b9c1-4b97-8f91-b64f3c75645d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:21.735967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735744 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:21.735967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.735757 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:21.737592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.737553 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials" pod "4aaa804a-b9c1-4b97-8f91-b64f3c75645d" (UID: "4aaa804a-b9c1-4b97-8f91-b64f3c75645d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:21.737697 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.737646 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk" pod "4aaa804a-b9c1-4b97-8f91-b64f3c75645d" (UID: "4aaa804a-b9c1-4b97-8f91-b64f3c75645d"). InnerVolumeSpecName "kube-api-access-hk9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:21.837993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.837965 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:21.837993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.837992 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk9dk\" (UniqueName: \"kubernetes.io/projected/4aaa804a-b9c1-4b97-8f91-b64f3c75645d-kube-api-access-hk9dk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:21.952424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.952347 6275 generic.go:356] "Generic (PLEG): container finished" podID="4aaa804a-b9c1-4b97-8f91-b64f3c75645d" containerID="21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d" exitCode=0 Oct 30 02:21:21.952570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.952537 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" event={"ID":"4aaa804a-b9c1-4b97-8f91-b64f3c75645d","Type":"ContainerDied","Data":"21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d"} Oct 30 02:21:21.952618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.952581 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" event={"ID":"4aaa804a-b9c1-4b97-8f91-b64f3c75645d","Type":"ContainerDied","Data":"2c47b52d7d22c65bd15af4935d3733ca11067ba420f78b179734178c4b481ee1"} Oct 30 02:21:21.952618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.952602 6275 scope.go:122] "RemoveContainer" containerID="21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d" Oct 30 02:21:21.952733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.952722 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565954b4c8-xdflc" Oct 30 02:21:21.955677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.955647 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" event={"ID":"d984fd98-a76a-4221-b04e-26ec74dd352a","Type":"ContainerStarted","Data":"2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f"} Oct 30 02:21:21.955822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.955699 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" event={"ID":"d984fd98-a76a-4221-b04e-26ec74dd352a","Type":"ContainerStarted","Data":"e34bcb019ec09cda1794a653ffed580f6753dc08b6029cb0f59a6fdb328c8aac"} Oct 30 02:21:21.955822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.955754 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:21.964094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.964075 6275 scope.go:122] "RemoveContainer" containerID="21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d" Oct 30 02:21:21.964377 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:21:21.964359 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d\": container with ID starting with 21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d not found: ID does not exist" containerID="21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d" Oct 30 02:21:21.964447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:21.964382 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d"} err="failed to get container status \"21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d\": rpc error: code = NotFound desc = could not find container \"21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d\": container with ID starting with 21464d008960b0c8c33d0b08581d78d7f46a360d6cd56058b192b4f49fb6299d not found: ID does not exist" Oct 30 02:21:22.013736 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:22.013711 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-565954b4c8-xdflc"] Oct 30 02:21:22.018929 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:22.018903 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-565954b4c8-xdflc"] Oct 30 02:21:22.063806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:22.063761 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" podStartSLOduration=2.063749318 podStartE2EDuration="2.063749318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:22.062691469 +0000 UTC m=+3110.454499103" watchObservedRunningTime="2026-10-30 02:21:22.063749318 +0000 UTC m=+3110.455556963" Oct 30 02:21:23.743334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:23.743306 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aaa804a-b9c1-4b97-8f91-b64f3c75645d" path="/var/lib/kubelet/pods/4aaa804a-b9c1-4b97-8f91-b64f3c75645d/volumes" Oct 30 02:21:24.945448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:24.945394 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz"] Oct 30 02:21:29.961704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:29.961478 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" podUID="d984fd98-a76a-4221-b04e-26ec74dd352a" containerName="payload-processing" containerID="cri-o://2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f" gracePeriod=25 Oct 30 02:21:30.283658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.283633 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:30.393574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.393543 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj\") pod \"d984fd98-a76a-4221-b04e-26ec74dd352a\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " Oct 30 02:21:30.393757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.393673 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials\") pod \"d984fd98-a76a-4221-b04e-26ec74dd352a\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " Oct 30 02:21:30.393757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.393740 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume\") pod \"d984fd98-a76a-4221-b04e-26ec74dd352a\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " Oct 30 02:21:30.393839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.393798 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume\") pod \"d984fd98-a76a-4221-b04e-26ec74dd352a\" (UID: \"d984fd98-a76a-4221-b04e-26ec74dd352a\") " Oct 30 02:21:30.394235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.394205 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume" pod "d984fd98-a76a-4221-b04e-26ec74dd352a" (UID: "d984fd98-a76a-4221-b04e-26ec74dd352a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:30.394391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.394231 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume" pod "d984fd98-a76a-4221-b04e-26ec74dd352a" (UID: "d984fd98-a76a-4221-b04e-26ec74dd352a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:30.394516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.394497 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.394589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.394516 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d984fd98-a76a-4221-b04e-26ec74dd352a-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.395909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.395876 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials" pod "d984fd98-a76a-4221-b04e-26ec74dd352a" (UID: "d984fd98-a76a-4221-b04e-26ec74dd352a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:30.396068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.396053 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj" pod "d984fd98-a76a-4221-b04e-26ec74dd352a" (UID: "d984fd98-a76a-4221-b04e-26ec74dd352a"). InnerVolumeSpecName "kube-api-access-fxskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:30.495510 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.495477 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.495510 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:30.495505 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxskj\" (UniqueName: \"kubernetes.io/projected/d984fd98-a76a-4221-b04e-26ec74dd352a-kube-api-access-fxskj\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:31.026854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.026815 6275 generic.go:356] "Generic (PLEG): container finished" podID="d984fd98-a76a-4221-b04e-26ec74dd352a" containerID="2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f" exitCode=0 Oct 30 02:21:31.027441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.026865 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" Oct 30 02:21:31.027441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.026892 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" event={"ID":"d984fd98-a76a-4221-b04e-26ec74dd352a","Type":"ContainerDied","Data":"2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f"} Oct 30 02:21:31.027441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.026944 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz" event={"ID":"d984fd98-a76a-4221-b04e-26ec74dd352a","Type":"ContainerDied","Data":"e34bcb019ec09cda1794a653ffed580f6753dc08b6029cb0f59a6fdb328c8aac"} Oct 30 02:21:31.027441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.026973 6275 scope.go:122] "RemoveContainer" containerID="2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f" Oct 30 02:21:31.046053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.046031 6275 scope.go:122] "RemoveContainer" containerID="2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f" Oct 30 02:21:31.046366 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:21:31.046344 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f\": container with ID starting with 2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f not found: ID does not exist" containerID="2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f" Oct 30 02:21:31.046446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.046381 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f"} err="failed to get container status \"2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f\": rpc error: code = NotFound desc = could not find container \"2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f\": container with ID starting with 2c703205d0c1a7ea8f152f846bf6bbbee9a92dac689a2cb16a8bb3be703fcd6f not found: ID does not exist" Oct 30 02:21:31.080600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.080566 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz"] Oct 30 02:21:31.087061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.087036 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-psvgz"] Oct 30 02:21:31.743511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:31.743481 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d984fd98-a76a-4221-b04e-26ec74dd352a" path="/var/lib/kubelet/pods/d984fd98-a76a-4221-b04e-26ec74dd352a/volumes" Oct 30 02:21:32.156036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.156002 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x"] Oct 30 02:21:32.156850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.156829 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d984fd98-a76a-4221-b04e-26ec74dd352a" containerName="payload-processing" Oct 30 02:21:32.156850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.156851 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d984fd98-a76a-4221-b04e-26ec74dd352a" containerName="payload-processing" Oct 30 02:21:32.156995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.156888 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4aaa804a-b9c1-4b97-8f91-b64f3c75645d" containerName="payload-processing" Oct 30 02:21:32.156995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.156894 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aaa804a-b9c1-4b97-8f91-b64f3c75645d" containerName="payload-processing" Oct 30 02:21:32.157067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.157032 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aaa804a-b9c1-4b97-8f91-b64f3c75645d" containerName="payload-processing" Oct 30 02:21:32.157067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.157049 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d984fd98-a76a-4221-b04e-26ec74dd352a" containerName="payload-processing" Oct 30 02:21:32.161196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.161172 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.179607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.179585 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-c8jmn\"" Oct 30 02:21:32.179742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.179729 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:21:32.205174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.190030 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2afd06b10b99\"" Oct 30 02:21:32.205174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.190090 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:21:32.205174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.190351 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:21:32.205174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.197006 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x"] Oct 30 02:21:32.226802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.226754 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.226977 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.226819 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.226977 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.226935 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.227305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.227282 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnm9\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.279608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.279573 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:21:32.334150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.330280 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnm9\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.334150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.330383 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.334150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.330424 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.334150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.330527 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.358199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.358038 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.359368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.359332 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.407246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.400206 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnm9\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.431064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.431028 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials\") pod \"payload-processing-external-model-5db9b8ddc4-mxg9x\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.477587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.477552 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:32.865684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:32.865662 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x"] Oct 30 02:21:32.868338 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:21:32.868311 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f60b33_957e_4e80_8d26_0003c311742f.slice/crio-b48a19eda93e907f856d25047406a8c65ee7afb4acc3ea9e3c870e8dbd5430fb WatchSource:0}: Error finding container b48a19eda93e907f856d25047406a8c65ee7afb4acc3ea9e3c870e8dbd5430fb: Status 404 returned error can't find the container with id b48a19eda93e907f856d25047406a8c65ee7afb4acc3ea9e3c870e8dbd5430fb Oct 30 02:21:33.110472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:33.110439 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" event={"ID":"b2f60b33-957e-4e80-8d26-0003c311742f","Type":"ContainerStarted","Data":"f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54"} Oct 30 02:21:33.110656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:33.110489 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" event={"ID":"b2f60b33-957e-4e80-8d26-0003c311742f","Type":"ContainerStarted","Data":"b48a19eda93e907f856d25047406a8c65ee7afb4acc3ea9e3c870e8dbd5430fb"} Oct 30 02:21:33.110705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:33.110666 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:33.171687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:33.171575 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" podStartSLOduration=1.171558203 podStartE2EDuration="1.171558203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:33.16546273 +0000 UTC m=+3121.557270366" watchObservedRunningTime="2026-10-30 02:21:33.171558203 +0000 UTC m=+3121.563365838" Oct 30 02:21:37.763438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:37.763395 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x"] Oct 30 02:21:37.767309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:37.767281 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:40.567020 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.566987 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h"] Oct 30 02:21:40.574758 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.574732 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.588695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.588665 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.588833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.588705 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.588833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.588803 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rk4\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.588957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.588943 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.607685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.607654 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h"] Oct 30 02:21:40.613276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.613242 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"routing-overlay\"" Oct 30 02:21:40.613453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.613365 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"kube-root-ca.crt\"" Oct 30 02:21:40.613453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.613242 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"openshift-service-ca.crt\"" Oct 30 02:21:40.613702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.613673 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-58fad4\"" Oct 30 02:21:40.648701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.648671 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"agc-cred-4777b568e245\"" Oct 30 02:21:40.649020 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.648993 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"payload-processing-external-model-e2e--4b9246f1-dockercfg-2dswk\"" Oct 30 02:21:40.690152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.690089 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.690310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.690262 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.690310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.690287 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.690466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.690319 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rk4\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.690788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.690764 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.690962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.690941 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.692724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.692695 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.712838 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.712814 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rk4\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:40.898298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:40.897390 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:41.397292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:41.397266 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h"] Oct 30 02:21:41.399704 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:21:41.399680 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b8f1f5_152a_4c93_b256_0fcba6c7caf6.slice/crio-dae70ff6b8fed383b82c88935e2d1b374e499240b0de4dc7af7a0bf2743edaf2 WatchSource:0}: Error finding container dae70ff6b8fed383b82c88935e2d1b374e499240b0de4dc7af7a0bf2743edaf2: Status 404 returned error can't find the container with id dae70ff6b8fed383b82c88935e2d1b374e499240b0de4dc7af7a0bf2743edaf2 Oct 30 02:21:41.934884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:41.934847 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw"] Oct 30 02:21:41.940236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:41.940215 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:41.971371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:41.971339 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4120adfa72e6\"" Oct 30 02:21:41.971559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:41.971458 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7n727\"" Oct 30 02:21:41.991265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:41.991219 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw"] Oct 30 02:21:42.134158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.125488 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.134158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.125541 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97mkn\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.134158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.125609 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.134158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.125785 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.228202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.227552 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.228202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.227737 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.228202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.227880 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.228202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.227920 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97mkn\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.229059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.229033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.231255 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.231195 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.241172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.241144 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.259841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.259809 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97mkn\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn\") pod \"payload-processing-external-model-7d7446f96-dg6pw\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.411192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.411149 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" event={"ID":"54b8f1f5-152a-4c93-b256-0fcba6c7caf6","Type":"ContainerStarted","Data":"1f54de9299c3c15755d7e21d8e2810317577c949568bcfb53a4a7665f906101e"} Oct 30 02:21:42.411387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.411221 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" event={"ID":"54b8f1f5-152a-4c93-b256-0fcba6c7caf6","Type":"ContainerStarted","Data":"dae70ff6b8fed383b82c88935e2d1b374e499240b0de4dc7af7a0bf2743edaf2"} Oct 30 02:21:42.411387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.411249 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:42.459769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.459717 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" podStartSLOduration=2.459702665 podStartE2EDuration="2.459702665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:42.454933251 +0000 UTC m=+3130.846740885" watchObservedRunningTime="2026-10-30 02:21:42.459702665 +0000 UTC m=+3130.851510336" Oct 30 02:21:42.557004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.556970 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:42.801277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.777228 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" podUID="b2f60b33-957e-4e80-8d26-0003c311742f" containerName="payload-processing" containerID="cri-o://f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54" gracePeriod=25 Oct 30 02:21:42.899524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:42.899496 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw"] Oct 30 02:21:42.919014 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:21:42.918974 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582468a5_2803_43a8_b07b_1604d658a6ed.slice/crio-6a34db2624cfcac381c29952593cb78bcc99e22c764a3bc8a49b5cdf226b5c8b WatchSource:0}: Error finding container 6a34db2624cfcac381c29952593cb78bcc99e22c764a3bc8a49b5cdf226b5c8b: Status 404 returned error can't find the container with id 6a34db2624cfcac381c29952593cb78bcc99e22c764a3bc8a49b5cdf226b5c8b Oct 30 02:21:43.344704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.344667 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:43.427519 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.427479 6275 generic.go:356] "Generic (PLEG): container finished" podID="b2f60b33-957e-4e80-8d26-0003c311742f" containerID="f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54" exitCode=0 Oct 30 02:21:43.427707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.427550 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" Oct 30 02:21:43.427707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.427565 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" event={"ID":"b2f60b33-957e-4e80-8d26-0003c311742f","Type":"ContainerDied","Data":"f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54"} Oct 30 02:21:43.427707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.427621 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x" event={"ID":"b2f60b33-957e-4e80-8d26-0003c311742f","Type":"ContainerDied","Data":"b48a19eda93e907f856d25047406a8c65ee7afb4acc3ea9e3c870e8dbd5430fb"} Oct 30 02:21:43.427707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.427653 6275 scope.go:122] "RemoveContainer" containerID="f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54" Oct 30 02:21:43.436165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.435856 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" event={"ID":"582468a5-2803-43a8-b07b-1604d658a6ed","Type":"ContainerStarted","Data":"80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3"} Oct 30 02:21:43.436165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.435920 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" event={"ID":"582468a5-2803-43a8-b07b-1604d658a6ed","Type":"ContainerStarted","Data":"6a34db2624cfcac381c29952593cb78bcc99e22c764a3bc8a49b5cdf226b5c8b"} Oct 30 02:21:43.436165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.435976 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:43.441620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.441600 6275 scope.go:122] "RemoveContainer" containerID="f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54" Oct 30 02:21:43.442018 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:21:43.441996 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54\": container with ID starting with f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54 not found: ID does not exist" containerID="f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54" Oct 30 02:21:43.442164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.442028 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54"} err="failed to get container status \"f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54\": rpc error: code = NotFound desc = could not find container \"f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54\": container with ID starting with f92375062250c9c27ba9616ed054e3270ddfb78fbfedafb2f65400e614186e54 not found: ID does not exist" Oct 30 02:21:43.455476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.455449 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume\") pod \"b2f60b33-957e-4e80-8d26-0003c311742f\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " Oct 30 02:21:43.455628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.455546 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume\") pod \"b2f60b33-957e-4e80-8d26-0003c311742f\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " Oct 30 02:21:43.455628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.455612 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials\") pod \"b2f60b33-957e-4e80-8d26-0003c311742f\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " Oct 30 02:21:43.455787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.455740 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9\") pod \"b2f60b33-957e-4e80-8d26-0003c311742f\" (UID: \"b2f60b33-957e-4e80-8d26-0003c311742f\") " Oct 30 02:21:43.457163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.456973 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume" pod "b2f60b33-957e-4e80-8d26-0003c311742f" (UID: "b2f60b33-957e-4e80-8d26-0003c311742f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:43.457163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.457057 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume" pod "b2f60b33-957e-4e80-8d26-0003c311742f" (UID: "b2f60b33-957e-4e80-8d26-0003c311742f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:43.458605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.458577 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9" pod "b2f60b33-957e-4e80-8d26-0003c311742f" (UID: "b2f60b33-957e-4e80-8d26-0003c311742f"). InnerVolumeSpecName "kube-api-access-7xnm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:43.458993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.458973 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials" pod "b2f60b33-957e-4e80-8d26-0003c311742f" (UID: "b2f60b33-957e-4e80-8d26-0003c311742f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:43.482995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.482927 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" podStartSLOduration=2.482908796 podStartE2EDuration="2.482908796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:43.480886723 +0000 UTC m=+3131.872694358" watchObservedRunningTime="2026-10-30 02:21:43.482908796 +0000 UTC m=+3131.874716431" Oct 30 02:21:43.558210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.558113 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:43.558210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.558159 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f60b33-957e-4e80-8d26-0003c311742f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:43.558210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.558172 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:43.558210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.558185 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnm9\" (UniqueName: \"kubernetes.io/projected/b2f60b33-957e-4e80-8d26-0003c311742f-kube-api-access-7xnm9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:43.790413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.790378 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x"] Oct 30 02:21:43.802535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:43.802497 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9b8ddc4-mxg9x"] Oct 30 02:21:45.770752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:45.770709 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f60b33-957e-4e80-8d26-0003c311742f" path="/var/lib/kubelet/pods/b2f60b33-957e-4e80-8d26-0003c311742f/volumes" Oct 30 02:21:46.713150 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:21:46.713078 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:21:46.850147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.844306 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf"] Oct 30 02:21:46.850147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.845748 6275 cpu_manager.go:414] "containerMap: removing container" podUID="b2f60b33-957e-4e80-8d26-0003c311742f" containerName="payload-processing" Oct 30 02:21:46.850147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.845774 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f60b33-957e-4e80-8d26-0003c311742f" containerName="payload-processing" Oct 30 02:21:46.850147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.845962 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f60b33-957e-4e80-8d26-0003c311742f" containerName="payload-processing" Oct 30 02:21:46.853789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.853027 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.888210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.888178 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"agc-cred-85320d861eea\"" Oct 30 02:21:46.889019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.888409 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whspg\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.889317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.889294 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.889481 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.889460 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.889654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.889636 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.897177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.896040 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf"] Oct 30 02:21:46.992768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.992664 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.993076 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.993031 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.993385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.993366 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.993826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.993806 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whspg\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.994454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.994422 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:46.995313 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:46.995283 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:47.004161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:47.001251 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:47.019133 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:47.019100 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whspg\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:47.187596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:47.187563 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:47.705160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:47.704086 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf"] Oct 30 02:21:48.721347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:48.721297 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" event={"ID":"0d432e06-0395-4d92-9758-4d76d190fbdc","Type":"ContainerStarted","Data":"d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a"} Oct 30 02:21:48.721799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:48.721368 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" event={"ID":"0d432e06-0395-4d92-9758-4d76d190fbdc","Type":"ContainerStarted","Data":"3bb6c37a28051e2be399fdd9eaf55de671046fb9f29641aeeb6e389918900e9f"} Oct 30 02:21:48.721799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:48.721402 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:48.768864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:48.768808 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" podStartSLOduration=2.768794583 podStartE2EDuration="2.768794583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:48.768189434 +0000 UTC m=+3137.159997066" watchObservedRunningTime="2026-10-30 02:21:48.768794583 +0000 UTC m=+3137.160602216" Oct 30 02:21:51.116200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:51.116164 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw"] Oct 30 02:21:51.119047 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:51.119023 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:53.439536 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:53.439497 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:21:56.140145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.139322 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" podUID="582468a5-2803-43a8-b07b-1604d658a6ed" containerName="payload-processing" containerID="cri-o://80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3" gracePeriod=25 Oct 30 02:21:56.497026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.496999 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:56.636143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.629469 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials\") pod \"582468a5-2803-43a8-b07b-1604d658a6ed\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " Oct 30 02:21:56.636143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.629539 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn\") pod \"582468a5-2803-43a8-b07b-1604d658a6ed\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " Oct 30 02:21:56.636143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.629657 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume\") pod \"582468a5-2803-43a8-b07b-1604d658a6ed\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " Oct 30 02:21:56.636143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.629863 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume\") pod \"582468a5-2803-43a8-b07b-1604d658a6ed\" (UID: \"582468a5-2803-43a8-b07b-1604d658a6ed\") " Oct 30 02:21:56.671884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.664409 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume" pod "582468a5-2803-43a8-b07b-1604d658a6ed" (UID: "582468a5-2803-43a8-b07b-1604d658a6ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:56.671884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.664550 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials" pod "582468a5-2803-43a8-b07b-1604d658a6ed" (UID: "582468a5-2803-43a8-b07b-1604d658a6ed"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:56.671884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.666207 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume" pod "582468a5-2803-43a8-b07b-1604d658a6ed" (UID: "582468a5-2803-43a8-b07b-1604d658a6ed"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:56.674798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.674766 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn" pod "582468a5-2803-43a8-b07b-1604d658a6ed" (UID: "582468a5-2803-43a8-b07b-1604d658a6ed"). InnerVolumeSpecName "kube-api-access-97mkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:56.732490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.732421 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:56.732490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.732446 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97mkn\" (UniqueName: \"kubernetes.io/projected/582468a5-2803-43a8-b07b-1604d658a6ed-kube-api-access-97mkn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:56.732490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.732457 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:56.732490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.732466 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/582468a5-2803-43a8-b07b-1604d658a6ed-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:21:56.926886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.926849 6275 generic.go:356] "Generic (PLEG): container finished" podID="582468a5-2803-43a8-b07b-1604d658a6ed" containerID="80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3" exitCode=0 Oct 30 02:21:56.927059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.926904 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" Oct 30 02:21:56.927059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.926934 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" event={"ID":"582468a5-2803-43a8-b07b-1604d658a6ed","Type":"ContainerDied","Data":"80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3"} Oct 30 02:21:56.927059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.927011 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw" event={"ID":"582468a5-2803-43a8-b07b-1604d658a6ed","Type":"ContainerDied","Data":"6a34db2624cfcac381c29952593cb78bcc99e22c764a3bc8a49b5cdf226b5c8b"} Oct 30 02:21:56.927059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.927038 6275 scope.go:122] "RemoveContainer" containerID="80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3" Oct 30 02:21:56.938331 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.938314 6275 scope.go:122] "RemoveContainer" containerID="80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3" Oct 30 02:21:56.938608 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:21:56.938589 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3\": container with ID starting with 80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3 not found: ID does not exist" containerID="80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3" Oct 30 02:21:56.938665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.938616 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3"} err="failed to get container status \"80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3\": rpc error: code = NotFound desc = could not find container \"80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3\": container with ID starting with 80814f6e4865fc4bc18f5891dc1d693affe1df1d7fdfdc4d25e86d57ee05d9c3 not found: ID does not exist" Oct 30 02:21:56.978859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.978827 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw"] Oct 30 02:21:56.988824 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:56.988755 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d7446f96-dg6pw"] Oct 30 02:21:57.137280 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.136701 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9"] Oct 30 02:21:57.138458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.138414 6275 cpu_manager.go:414] "containerMap: removing container" podUID="582468a5-2803-43a8-b07b-1604d658a6ed" containerName="payload-processing" Oct 30 02:21:57.138458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.138448 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582468a5-2803-43a8-b07b-1604d658a6ed" containerName="payload-processing" Oct 30 02:21:57.138667 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.138640 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="582468a5-2803-43a8-b07b-1604d658a6ed" containerName="payload-processing" Oct 30 02:21:57.144412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.144383 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.163653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.163623 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9"] Oct 30 02:21:57.163802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.163668 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"routing-overlay\"" Oct 30 02:21:57.168784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.168756 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"agc-cred-a71d0b7b9515\"" Oct 30 02:21:57.169060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.168805 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-e2e--74e737ad-dockercfg-mbn24\"" Oct 30 02:21:57.241992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.241908 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.242159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.242057 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.242159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.242084 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.242247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.242178 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lsp\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.264532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.264499 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-000edd\"" Oct 30 02:21:57.264714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.264695 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"kube-root-ca.crt\"" Oct 30 02:21:57.264896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.264881 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"openshift-service-ca.crt\"" Oct 30 02:21:57.347745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.347709 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.355580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.355549 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.359901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.359865 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lsp\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.360782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.360754 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.361583 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.359719 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.375599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.348731 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.397068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.397033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lsp\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.430581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.430548 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.464262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.463196 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.742596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.742556 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582468a5-2803-43a8-b07b-1604d658a6ed" path="/var/lib/kubelet/pods/582468a5-2803-43a8-b07b-1604d658a6ed/volumes" Oct 30 02:21:57.783225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.783194 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9"] Oct 30 02:21:57.785474 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:21:57.785447 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15351072_7614_4611_9de9_bcaabf3310ab.slice/crio-714858346fa8d7b6090c06d07c7bfacbc146c292fe1c31fee4eb8bbd7036f3f5 WatchSource:0}: Error finding container 714858346fa8d7b6090c06d07c7bfacbc146c292fe1c31fee4eb8bbd7036f3f5: Status 404 returned error can't find the container with id 714858346fa8d7b6090c06d07c7bfacbc146c292fe1c31fee4eb8bbd7036f3f5 Oct 30 02:21:57.936470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.936430 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" event={"ID":"15351072-7614-4611-9de9-bcaabf3310ab","Type":"ContainerStarted","Data":"b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6"} Oct 30 02:21:57.936638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.936495 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:21:57.936638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.936522 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" event={"ID":"15351072-7614-4611-9de9-bcaabf3310ab","Type":"ContainerStarted","Data":"714858346fa8d7b6090c06d07c7bfacbc146c292fe1c31fee4eb8bbd7036f3f5"} Oct 30 02:21:57.980557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:57.980510 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" podStartSLOduration=0.980498482 podStartE2EDuration="980.498482ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:57.976974626 +0000 UTC m=+3146.368782260" watchObservedRunningTime="2026-10-30 02:21:57.980498482 +0000 UTC m=+3146.372306138" Oct 30 02:21:59.741770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:59.741741 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:21:59.821592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:21:59.821548 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h"] Oct 30 02:22:02.460315 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.458053 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c"] Oct 30 02:22:02.463452 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.463426 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.476988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.476910 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c"] Oct 30 02:22:02.479027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.478996 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.479246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.479065 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.479246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.479200 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.479371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.479297 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsql\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.480588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.480204 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"kube-root-ca.crt\"" Oct 30 02:22:02.480588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.480405 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"payload-processing-external-model-e2e--6aaede7f-dockercfg-j4qbl\"" Oct 30 02:22:02.480588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.480499 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-95f40f\"" Oct 30 02:22:02.480588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.480557 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"openshift-service-ca.crt\"" Oct 30 02:22:02.488308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.488199 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"routing-overlay\"" Oct 30 02:22:02.488443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.488310 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"agc-cred-43923dcebdf1\"" Oct 30 02:22:02.580666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.580625 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.580841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.580681 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.580841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.580709 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.580841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.580773 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsql\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.581520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.581493 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.581672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.581650 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.583174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.583155 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.602248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.602224 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsql\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:02.797434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:02.796184 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:03.147458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:03.147340 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c"] Oct 30 02:22:04.017137 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:04.017089 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" event={"ID":"0fc5e8cc-b7ac-48fa-8c45-28138503d769","Type":"ContainerStarted","Data":"54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6"} Oct 30 02:22:04.017567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:04.017187 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:04.017567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:04.017212 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" event={"ID":"0fc5e8cc-b7ac-48fa-8c45-28138503d769","Type":"ContainerStarted","Data":"76c55a17e074c0e366893cc72c48b40ab60d50306af9d53ff6d7ed4650e0f6d0"} Oct 30 02:22:04.058156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:04.058090 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" podStartSLOduration=2.058076963 podStartE2EDuration="2.058076963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:04.055178632 +0000 UTC m=+3152.446986259" watchObservedRunningTime="2026-10-30 02:22:04.058076963 +0000 UTC m=+3152.449884646" Oct 30 02:22:04.822353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:04.822312 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" podUID="54b8f1f5-152a-4c93-b256-0fcba6c7caf6" containerName="payload-processing" containerID="cri-o://1f54de9299c3c15755d7e21d8e2810317577c949568bcfb53a4a7665f906101e" gracePeriod=25 Oct 30 02:22:05.038015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.037968 6275 generic.go:356] "Generic (PLEG): container finished" podID="54b8f1f5-152a-4c93-b256-0fcba6c7caf6" containerID="1f54de9299c3c15755d7e21d8e2810317577c949568bcfb53a4a7665f906101e" exitCode=0 Oct 30 02:22:05.038565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.038383 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" event={"ID":"54b8f1f5-152a-4c93-b256-0fcba6c7caf6","Type":"ContainerDied","Data":"1f54de9299c3c15755d7e21d8e2810317577c949568bcfb53a4a7665f906101e"} Oct 30 02:22:05.111941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.111913 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:22:05.227098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.227063 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4\") pod \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " Oct 30 02:22:05.227306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.227118 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume\") pod \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " Oct 30 02:22:05.227357 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.227344 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials\") pod \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " Oct 30 02:22:05.227481 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.227465 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume\") pod \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\" (UID: \"54b8f1f5-152a-4c93-b256-0fcba6c7caf6\") " Oct 30 02:22:05.227602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.227581 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume" pod "54b8f1f5-152a-4c93-b256-0fcba6c7caf6" (UID: "54b8f1f5-152a-4c93-b256-0fcba6c7caf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:05.227865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.227832 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume" pod "54b8f1f5-152a-4c93-b256-0fcba6c7caf6" (UID: "54b8f1f5-152a-4c93-b256-0fcba6c7caf6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:05.228196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.228178 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:05.228196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.228197 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:05.229589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.229564 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials" pod "54b8f1f5-152a-4c93-b256-0fcba6c7caf6" (UID: "54b8f1f5-152a-4c93-b256-0fcba6c7caf6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:05.229692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.229605 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4" pod "54b8f1f5-152a-4c93-b256-0fcba6c7caf6" (UID: "54b8f1f5-152a-4c93-b256-0fcba6c7caf6"). InnerVolumeSpecName "kube-api-access-44rk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:05.329726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.329682 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:05.329726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:05.329721 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44rk4\" (UniqueName: \"kubernetes.io/projected/54b8f1f5-152a-4c93-b256-0fcba6c7caf6-kube-api-access-44rk4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.047367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:06.047338 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" Oct 30 02:22:06.047857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:06.047372 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h" event={"ID":"54b8f1f5-152a-4c93-b256-0fcba6c7caf6","Type":"ContainerDied","Data":"dae70ff6b8fed383b82c88935e2d1b374e499240b0de4dc7af7a0bf2743edaf2"} Oct 30 02:22:06.047857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:06.047441 6275 scope.go:122] "RemoveContainer" containerID="1f54de9299c3c15755d7e21d8e2810317577c949568bcfb53a4a7665f906101e" Oct 30 02:22:06.094899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:06.094867 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h"] Oct 30 02:22:06.104843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:06.104819 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdw8k7h"] Oct 30 02:22:07.741665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:07.741632 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b8f1f5-152a-4c93-b256-0fcba6c7caf6" path="/var/lib/kubelet/pods/54b8f1f5-152a-4c93-b256-0fcba6c7caf6/volumes" Oct 30 02:22:08.948563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:08.948535 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:22:12.805108 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.805028 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4"] Oct 30 02:22:12.805927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.805908 6275 cpu_manager.go:414] "containerMap: removing container" podUID="54b8f1f5-152a-4c93-b256-0fcba6c7caf6" containerName="payload-processing" Oct 30 02:22:12.805927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.805928 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b8f1f5-152a-4c93-b256-0fcba6c7caf6" containerName="payload-processing" Oct 30 02:22:12.806074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.806064 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b8f1f5-152a-4c93-b256-0fcba6c7caf6" containerName="payload-processing" Oct 30 02:22:12.811471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.811448 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.860026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.859988 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5v54n\"" Oct 30 02:22:12.862180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.861747 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.862423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.862401 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.862756 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.862732 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.862888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.862873 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57k6q\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.863019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.863006 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.874825 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.874789 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.876061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.876036 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.876425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.876400 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.876600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.876582 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.880197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.880175 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4"] Oct 30 02:22:12.978660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978621 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.978834 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978674 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.978834 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978770 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.978834 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978800 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57k6q\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978854 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978916 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.978945 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979078 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979110 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979331 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979219 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979517 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979494 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979525 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979572 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.979636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.979581 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.981446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.981423 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:12.981579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:12.981561 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:13.004551 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.004515 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:13.005587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.005566 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57k6q\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:13.126648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.126569 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:13.461265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.461236 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4"] Oct 30 02:22:13.463184 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:22:13.463158 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f84fdca_7562_4edd_acc3_cf9f7c6cb183.slice/crio-4284675f2816e5f3ee5464f2202bb390ee736b3bea4fb2dfbda261e60cf06da8 WatchSource:0}: Error finding container 4284675f2816e5f3ee5464f2202bb390ee736b3bea4fb2dfbda261e60cf06da8: Status 404 returned error can't find the container with id 4284675f2816e5f3ee5464f2202bb390ee736b3bea4fb2dfbda261e60cf06da8 Oct 30 02:22:13.466215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.466174 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:22:13.466354 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.466249 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:22:13.466354 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:13.466278 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:22:14.113651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:14.113605 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" event={"ID":"9f84fdca-7562-4edd-acc3-cf9f7c6cb183","Type":"ContainerStarted","Data":"5afd46686a9e948c881b940274c2fdc9da528f1666840ca90649c900a3ab7ee9"} Oct 30 02:22:14.114084 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:14.113674 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" event={"ID":"9f84fdca-7562-4edd-acc3-cf9f7c6cb183","Type":"ContainerStarted","Data":"4284675f2816e5f3ee5464f2202bb390ee736b3bea4fb2dfbda261e60cf06da8"} Oct 30 02:22:14.126724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:14.126696 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:14.132679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:14.132659 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:14.169444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:14.169383 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" podStartSLOduration=2.169367041 podStartE2EDuration="2.169367041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:14.166710917 +0000 UTC m=+3162.558518550" watchObservedRunningTime="2026-10-30 02:22:14.169367041 +0000 UTC m=+3162.561174675" Oct 30 02:22:15.041767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:15.041737 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:15.124306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:15.124263 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:15.125272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:15.125245 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:22:15.657060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:15.657018 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c"] Oct 30 02:22:16.698789 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:16.698748 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:22:20.658265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:20.658219 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" podUID="0fc5e8cc-b7ac-48fa-8c45-28138503d769" containerName="payload-processing" containerID="cri-o://54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6" gracePeriod=25 Oct 30 02:22:20.950872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:20.950839 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:21.019468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.019433 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials\") pod \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " Oct 30 02:22:21.019691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.019676 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume\") pod \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " Oct 30 02:22:21.019774 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.019724 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql\") pod \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " Oct 30 02:22:21.019837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.019805 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume\") pod \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\" (UID: \"0fc5e8cc-b7ac-48fa-8c45-28138503d769\") " Oct 30 02:22:21.020392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.020176 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume" pod "0fc5e8cc-b7ac-48fa-8c45-28138503d769" (UID: "0fc5e8cc-b7ac-48fa-8c45-28138503d769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:21.020610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.020453 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:21.020610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.020575 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume" pod "0fc5e8cc-b7ac-48fa-8c45-28138503d769" (UID: "0fc5e8cc-b7ac-48fa-8c45-28138503d769"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:21.022776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.022744 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql" pod "0fc5e8cc-b7ac-48fa-8c45-28138503d769" (UID: "0fc5e8cc-b7ac-48fa-8c45-28138503d769"). InnerVolumeSpecName "kube-api-access-djsql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:21.022984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.022959 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials" pod "0fc5e8cc-b7ac-48fa-8c45-28138503d769" (UID: "0fc5e8cc-b7ac-48fa-8c45-28138503d769"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:21.121790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.121746 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5e8cc-b7ac-48fa-8c45-28138503d769-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:21.121790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.121777 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:21.121790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.121788 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djsql\" (UniqueName: \"kubernetes.io/projected/0fc5e8cc-b7ac-48fa-8c45-28138503d769-kube-api-access-djsql\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:21.194292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.194210 6275 generic.go:356] "Generic (PLEG): container finished" podID="0fc5e8cc-b7ac-48fa-8c45-28138503d769" containerID="54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6" exitCode=0 Oct 30 02:22:21.194292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.194260 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" Oct 30 02:22:21.194545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.194293 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" event={"ID":"0fc5e8cc-b7ac-48fa-8c45-28138503d769","Type":"ContainerDied","Data":"54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6"} Oct 30 02:22:21.194545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.194360 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c" event={"ID":"0fc5e8cc-b7ac-48fa-8c45-28138503d769","Type":"ContainerDied","Data":"76c55a17e074c0e366893cc72c48b40ab60d50306af9d53ff6d7ed4650e0f6d0"} Oct 30 02:22:21.194545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.194391 6275 scope.go:122] "RemoveContainer" containerID="54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6" Oct 30 02:22:21.208435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.208409 6275 scope.go:122] "RemoveContainer" containerID="54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6" Oct 30 02:22:21.208770 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:21.208746 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6\": container with ID starting with 54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6 not found: ID does not exist" containerID="54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6" Oct 30 02:22:21.208850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.208780 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6"} err="failed to get container status \"54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6\": rpc error: code = NotFound desc = could not find container \"54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6\": container with ID starting with 54e5a8e5db9c6cf9e51e0124af78b4be6ed935019f3f27d62745e9b50a1bacc6 not found: ID does not exist" Oct 30 02:22:21.242104 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.242043 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c"] Oct 30 02:22:21.246843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.246816 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85vmx8c"] Oct 30 02:22:21.741715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:21.741682 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc5e8cc-b7ac-48fa-8c45-28138503d769" path="/var/lib/kubelet/pods/0fc5e8cc-b7ac-48fa-8c45-28138503d769/volumes" Oct 30 02:22:33.687491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:33.687456 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9"] Oct 30 02:22:35.198397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.198362 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp"] Oct 30 02:22:35.199797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.199768 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0fc5e8cc-b7ac-48fa-8c45-28138503d769" containerName="payload-processing" Oct 30 02:22:35.199797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.199796 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc5e8cc-b7ac-48fa-8c45-28138503d769" containerName="payload-processing" Oct 30 02:22:35.200057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.200041 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc5e8cc-b7ac-48fa-8c45-28138503d769" containerName="payload-processing" Oct 30 02:22:35.206321 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.206183 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.228143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.220427 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"agc-cred-2a752186c169\"" Oct 30 02:22:35.232442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.232409 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp"] Oct 30 02:22:35.232752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.232447 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-e2e--b7f31742-dockercfg-gjv6n\"" Oct 30 02:22:35.292184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.292024 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcw5\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.292184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.292112 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.292464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.292238 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.292464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.292269 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.428830 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.428772 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcw5\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.429174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.429139 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.429416 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.429386 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.429562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.429544 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.430429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.430397 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.430552 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.430518 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:22:35.430552 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.430533 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp: secret "agc-cred-2a752186c169" not found Oct 30 02:22:35.430675 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.430587 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials podName:42083274-6f9c-4d62-bb18-c8dd8eedecf8 nodeName:}" failed. No retries permitted until 2026-10-30 02:22:35.930568835 +0000 UTC m=+3184.322376460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" (UID: "42083274-6f9c-4d62-bb18-c8dd8eedecf8") : secret "agc-cred-2a752186c169" not found Oct 30 02:22:35.430990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.430957 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.515141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.509439 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcw5\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.587925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.587843 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp"] Oct 30 02:22:35.588292 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.588269 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" podUID="42083274-6f9c-4d62-bb18-c8dd8eedecf8" Oct 30 02:22:35.945467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:35.945376 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:35.946272 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.945784 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:22:35.946412 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.946399 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp: secret "agc-cred-2a752186c169" not found Oct 30 02:22:35.946579 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:35.946567 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials podName:42083274-6f9c-4d62-bb18-c8dd8eedecf8 nodeName:}" failed. No retries permitted until 2026-10-30 02:22:36.946541749 +0000 UTC m=+3185.338349375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" (UID: "42083274-6f9c-4d62-bb18-c8dd8eedecf8") : secret "agc-cred-2a752186c169" not found Oct 30 02:22:36.499921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.499877 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:36.509181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.509158 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:36.562004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.561975 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume\") pod \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " Oct 30 02:22:36.562219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.562025 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume\") pod \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " Oct 30 02:22:36.562219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.562063 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5\") pod \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " Oct 30 02:22:36.562497 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.562476 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume" pod "42083274-6f9c-4d62-bb18-c8dd8eedecf8" (UID: "42083274-6f9c-4d62-bb18-c8dd8eedecf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:36.562549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.562485 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume" pod "42083274-6f9c-4d62-bb18-c8dd8eedecf8" (UID: "42083274-6f9c-4d62-bb18-c8dd8eedecf8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:36.562880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.562866 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:36.562915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.562883 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42083274-6f9c-4d62-bb18-c8dd8eedecf8-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:36.564328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.564308 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5" pod "42083274-6f9c-4d62-bb18-c8dd8eedecf8" (UID: "42083274-6f9c-4d62-bb18-c8dd8eedecf8"). InnerVolumeSpecName "kube-api-access-6lcw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:36.664554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.664525 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lcw5\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-kube-api-access-6lcw5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:36.969601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:36.969566 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp\" (UID: \"42083274-6f9c-4d62-bb18-c8dd8eedecf8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:36.969797 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:36.969725 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:22:36.969797 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:36.969746 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp: secret "agc-cred-2a752186c169" not found Oct 30 02:22:36.969885 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:36.969809 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials podName:42083274-6f9c-4d62-bb18-c8dd8eedecf8 nodeName:}" failed. No retries permitted until 2026-10-30 02:22:38.969794764 +0000 UTC m=+3187.361602390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" (UID: "42083274-6f9c-4d62-bb18-c8dd8eedecf8") : secret "agc-cred-2a752186c169" not found Oct 30 02:22:37.508162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:37.507972 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp" Oct 30 02:22:37.582218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:37.582181 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp"] Oct 30 02:22:37.591505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:37.591472 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bvw7pp"] Oct 30 02:22:37.684474 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:37.684443 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42083274-6f9c-4d62-bb18-c8dd8eedecf8-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:37.740955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:37.740925 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42083274-6f9c-4d62-bb18-c8dd8eedecf8" path="/var/lib/kubelet/pods/42083274-6f9c-4d62-bb18-c8dd8eedecf8/volumes" Oct 30 02:22:38.688348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:38.688307 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" podUID="45ec5129-19be-4b66-a957-10cc7825e237" containerName="payload-processing" containerID="cri-o://1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470" gracePeriod=25 Oct 30 02:22:39.029479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.029453 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:22:39.138789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.138760 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume\") pod \"45ec5129-19be-4b66-a957-10cc7825e237\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " Oct 30 02:22:39.138958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.138933 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials\") pod \"45ec5129-19be-4b66-a957-10cc7825e237\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " Oct 30 02:22:39.139043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.139030 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume\") pod \"45ec5129-19be-4b66-a957-10cc7825e237\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " Oct 30 02:22:39.139112 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.139100 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z\") pod \"45ec5129-19be-4b66-a957-10cc7825e237\" (UID: \"45ec5129-19be-4b66-a957-10cc7825e237\") " Oct 30 02:22:39.139194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.139152 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume" pod "45ec5129-19be-4b66-a957-10cc7825e237" (UID: "45ec5129-19be-4b66-a957-10cc7825e237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:39.139422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.139402 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume" pod "45ec5129-19be-4b66-a957-10cc7825e237" (UID: "45ec5129-19be-4b66-a957-10cc7825e237"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:39.139666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.139649 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:39.139729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.139672 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ec5129-19be-4b66-a957-10cc7825e237-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:39.141258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.141236 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z" pod "45ec5129-19be-4b66-a957-10cc7825e237" (UID: "45ec5129-19be-4b66-a957-10cc7825e237"). InnerVolumeSpecName "kube-api-access-g7z5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:39.141340 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.141326 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials" pod "45ec5129-19be-4b66-a957-10cc7825e237" (UID: "45ec5129-19be-4b66-a957-10cc7825e237"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:39.240845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.240807 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:39.240845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.240837 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7z5z\" (UniqueName: \"kubernetes.io/projected/45ec5129-19be-4b66-a957-10cc7825e237-kube-api-access-g7z5z\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:22:39.522722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.522686 6275 generic.go:356] "Generic (PLEG): container finished" podID="45ec5129-19be-4b66-a957-10cc7825e237" containerID="1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470" exitCode=0 Oct 30 02:22:39.522887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.522739 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" Oct 30 02:22:39.522887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.522775 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" event={"ID":"45ec5129-19be-4b66-a957-10cc7825e237","Type":"ContainerDied","Data":"1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470"} Oct 30 02:22:39.522887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.522831 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9" event={"ID":"45ec5129-19be-4b66-a957-10cc7825e237","Type":"ContainerDied","Data":"14db95e623378722ea68c48caac9776b25d76f2d8febd64cec96afadaff70b83"} Oct 30 02:22:39.522887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.522859 6275 scope.go:122] "RemoveContainer" containerID="1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470" Oct 30 02:22:39.534001 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.533979 6275 scope.go:122] "RemoveContainer" containerID="1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470" Oct 30 02:22:39.534299 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:39.534279 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470\": container with ID starting with 1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470 not found: ID does not exist" containerID="1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470" Oct 30 02:22:39.534387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.534312 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470"} err="failed to get container status \"1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470\": rpc error: code = NotFound desc = could not find container \"1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470\": container with ID starting with 1b33eb20de1a9be845c9b30eef9ad164261037e76a7c4dc83ff07a8ee5b98470 not found: ID does not exist" Oct 30 02:22:39.572100 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.572066 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9"] Oct 30 02:22:39.582273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.582247 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-8488qzjr9"] Oct 30 02:22:39.741097 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:39.741062 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ec5129-19be-4b66-a957-10cc7825e237" path="/var/lib/kubelet/pods/45ec5129-19be-4b66-a957-10cc7825e237/volumes" Oct 30 02:22:46.697810 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:22:46.697772 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:22:52.510835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.510790 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp"] Oct 30 02:22:52.511630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.511613 6275 cpu_manager.go:414] "containerMap: removing container" podUID="45ec5129-19be-4b66-a957-10cc7825e237" containerName="payload-processing" Oct 30 02:22:52.511630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.511630 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ec5129-19be-4b66-a957-10cc7825e237" containerName="payload-processing" Oct 30 02:22:52.511784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.511756 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ec5129-19be-4b66-a957-10cc7825e237" containerName="payload-processing" Oct 30 02:22:52.521139 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.517528 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.538873 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.538844 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp"] Oct 30 02:22:52.543007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.542981 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:22:52.543589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.543335 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:22:52.543771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.543496 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:22:52.543771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.543547 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:22:52.543771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.543591 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-413fa06e967f\"" Oct 30 02:22:52.546896 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.546875 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-j7lcj\"" Oct 30 02:22:52.556195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.556102 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.556609 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.556590 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.556839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.556821 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.557071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.557055 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z5q\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.710147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.709703 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.710147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.709771 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.710147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.709875 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z5q\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.710147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.710055 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.712156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.711191 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.712156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.711774 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.729857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.721168 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.789196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.783809 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z5q\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q\") pod \"payload-processing-external-model-574c95cfc5-p2vlp\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:52.838147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:52.837551 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:53.293431 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:53.293404 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp"] Oct 30 02:22:53.300333 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:22:53.300299 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4239de61_dff5_4d11_b232_6728936cc8b5.slice/crio-e7a7bb0b5876a1f73524a8074cf2652e397538ef538b9bf7ff4a08353ffd063a WatchSource:0}: Error finding container e7a7bb0b5876a1f73524a8074cf2652e397538ef538b9bf7ff4a08353ffd063a: Status 404 returned error can't find the container with id e7a7bb0b5876a1f73524a8074cf2652e397538ef538b9bf7ff4a08353ffd063a Oct 30 02:22:53.758798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:53.758766 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" event={"ID":"4239de61-dff5-4d11-b232-6728936cc8b5","Type":"ContainerStarted","Data":"3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea"} Oct 30 02:22:53.759215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:53.758829 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" event={"ID":"4239de61-dff5-4d11-b232-6728936cc8b5","Type":"ContainerStarted","Data":"e7a7bb0b5876a1f73524a8074cf2652e397538ef538b9bf7ff4a08353ffd063a"} Oct 30 02:22:53.759215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:53.758861 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:22:53.799446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:22:53.799393 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" podStartSLOduration=1.799379571 podStartE2EDuration="1.799379571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:53.798055987 +0000 UTC m=+3202.189863620" watchObservedRunningTime="2026-10-30 02:22:53.799379571 +0000 UTC m=+3202.191187203" Oct 30 02:23:04.770317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:04.770279 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:23:16.703192 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:23:16.703136 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:23:18.226248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.226209 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l"] Oct 30 02:23:18.231229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.231205 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.275680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.275194 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"openshift-service-ca.crt\"" Oct 30 02:23:18.275680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.275337 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-e2e--1bd5e061-dockercfg-jz6pw\"" Oct 30 02:23:18.275680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.275431 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"kube-root-ca.crt\"" Oct 30 02:23:18.275680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.275493 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a0fc73\"" Oct 30 02:23:18.275680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.275552 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"agc-cred-80a3dc502ca9\"" Oct 30 02:23:18.275680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.275637 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"routing-overlay\"" Oct 30 02:23:18.282599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.282577 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l"] Oct 30 02:23:18.325919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.325161 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.325919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.325259 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.325919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.325454 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmh9\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.325919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.325605 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.426810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.426771 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.426984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.426834 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.426984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.426879 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.427063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.426993 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmh9\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.427869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.427841 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.428400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.428376 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.429913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.429888 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.468779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.468750 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmh9\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.577524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.577490 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:18.938477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:18.920863 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l"] Oct 30 02:23:19.489206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:19.464465 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l"] Oct 30 02:23:20.250034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:20.249974 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" event={"ID":"ef48d546-5a62-4a27-b47f-d388b4c000e5","Type":"ContainerStarted","Data":"6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37"} Oct 30 02:23:20.250260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:20.250097 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" event={"ID":"ef48d546-5a62-4a27-b47f-d388b4c000e5","Type":"ContainerStarted","Data":"4aae69f00f27987d14df75c9da31b7731b490f8aefe6a34f268cda7e05d4c98a"} Oct 30 02:23:20.250260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:20.250197 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:20.290156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:20.289877 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" podStartSLOduration=2.289858008 podStartE2EDuration="2.289858008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:20.287758177 +0000 UTC m=+3228.679565802" watchObservedRunningTime="2026-10-30 02:23:20.289858008 +0000 UTC m=+3228.681665633" Oct 30 02:23:25.250984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.250926 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" podUID="ef48d546-5a62-4a27-b47f-d388b4c000e5" containerName="payload-processing" containerID="cri-o://6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37" gracePeriod=25 Oct 30 02:23:25.526717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.526686 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:25.569984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.569945 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials\") pod \"ef48d546-5a62-4a27-b47f-d388b4c000e5\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " Oct 30 02:23:25.570438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.570374 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume\") pod \"ef48d546-5a62-4a27-b47f-d388b4c000e5\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " Oct 30 02:23:25.570589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.570571 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9\") pod \"ef48d546-5a62-4a27-b47f-d388b4c000e5\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " Oct 30 02:23:25.570711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.570683 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume\") pod \"ef48d546-5a62-4a27-b47f-d388b4c000e5\" (UID: \"ef48d546-5a62-4a27-b47f-d388b4c000e5\") " Oct 30 02:23:25.570869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.570703 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume" pod "ef48d546-5a62-4a27-b47f-d388b4c000e5" (UID: "ef48d546-5a62-4a27-b47f-d388b4c000e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:25.571472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.571076 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume" pod "ef48d546-5a62-4a27-b47f-d388b4c000e5" (UID: "ef48d546-5a62-4a27-b47f-d388b4c000e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:25.571472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.571427 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:25.571472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.571450 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef48d546-5a62-4a27-b47f-d388b4c000e5-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:25.573439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.573416 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials" pod "ef48d546-5a62-4a27-b47f-d388b4c000e5" (UID: "ef48d546-5a62-4a27-b47f-d388b4c000e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:25.573582 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.573528 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9" pod "ef48d546-5a62-4a27-b47f-d388b4c000e5" (UID: "ef48d546-5a62-4a27-b47f-d388b4c000e5"). InnerVolumeSpecName "kube-api-access-8kmh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:25.672630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.672600 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:25.672630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:25.672627 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kmh9\" (UniqueName: \"kubernetes.io/projected/ef48d546-5a62-4a27-b47f-d388b4c000e5-kube-api-access-8kmh9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:26.321487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.321444 6275 generic.go:356] "Generic (PLEG): container finished" podID="ef48d546-5a62-4a27-b47f-d388b4c000e5" containerID="6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37" exitCode=0 Oct 30 02:23:26.321988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.321498 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" Oct 30 02:23:26.321988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.321523 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" event={"ID":"ef48d546-5a62-4a27-b47f-d388b4c000e5","Type":"ContainerDied","Data":"6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37"} Oct 30 02:23:26.321988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.321609 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l" event={"ID":"ef48d546-5a62-4a27-b47f-d388b4c000e5","Type":"ContainerDied","Data":"4aae69f00f27987d14df75c9da31b7731b490f8aefe6a34f268cda7e05d4c98a"} Oct 30 02:23:26.321988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.321934 6275 scope.go:122] "RemoveContainer" containerID="6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37" Oct 30 02:23:26.332870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.332842 6275 scope.go:122] "RemoveContainer" containerID="6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37" Oct 30 02:23:26.333142 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:23:26.333108 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37\": container with ID starting with 6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37 not found: ID does not exist" containerID="6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37" Oct 30 02:23:26.333236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.333155 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37"} err="failed to get container status \"6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37\": rpc error: code = NotFound desc = could not find container \"6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37\": container with ID starting with 6cda743c0d76c92025271f73db0c055419bdbeeb890e559c3b15964201c74f37 not found: ID does not exist" Oct 30 02:23:26.377025 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.376992 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l"] Oct 30 02:23:26.380617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:26.380591 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-847745v2l"] Oct 30 02:23:27.741393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:27.741358 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef48d546-5a62-4a27-b47f-d388b4c000e5" path="/var/lib/kubelet/pods/ef48d546-5a62-4a27-b47f-d388b4c000e5/volumes" Oct 30 02:23:32.334750 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:32.334711 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp"] Oct 30 02:23:35.074838 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.074802 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4"] Oct 30 02:23:35.075656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.075641 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ef48d546-5a62-4a27-b47f-d388b4c000e5" containerName="payload-processing" Oct 30 02:23:35.075705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.075658 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef48d546-5a62-4a27-b47f-d388b4c000e5" containerName="payload-processing" Oct 30 02:23:35.075793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.075783 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef48d546-5a62-4a27-b47f-d388b4c000e5" containerName="payload-processing" Oct 30 02:23:35.079783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.079697 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.095369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.095335 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3d5da6-openshift-default-dockercfg-t75nk\"" Oct 30 02:23:35.112102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.112070 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4"] Oct 30 02:23:35.174342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174308 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174360 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174463 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174500 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174519 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174687 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgx8f\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174753 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174782 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.174900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.174888 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285148 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx8f\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285271 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285622 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285776 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285850 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285905 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.285979 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.286033 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.293095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.286056 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.298404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.286596 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.322229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.297962 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.322460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.300833 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.322560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.310419 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.331238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.301148 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.340775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.340702 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.367143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.346380 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.367392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.351346 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.367513 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.363786 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx8f\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.401843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.401813 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:35.949999 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.949904 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4"] Oct 30 02:23:35.953001 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:23:35.952956 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8def214d_3691_4efe_97fb_482d22899207.slice/crio-f9810b1edbf69df346a3b85d0b3804c8f8c09d11fdc4dc5dd83daceaa9dc5e9e WatchSource:0}: Error finding container f9810b1edbf69df346a3b85d0b3804c8f8c09d11fdc4dc5dd83daceaa9dc5e9e: Status 404 returned error can't find the container with id f9810b1edbf69df346a3b85d0b3804c8f8c09d11fdc4dc5dd83daceaa9dc5e9e Oct 30 02:23:35.957106 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.957062 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:23:35.957234 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.957200 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:23:35.957339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:35.957253 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:23:36.421109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:36.421066 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" event={"ID":"8def214d-3691-4efe-97fb-482d22899207","Type":"ContainerStarted","Data":"79c0b88f753f0038851b4f01a63ab9e72da4748ca7b6669189831639564e6ada"} Oct 30 02:23:36.421654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:36.421150 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" event={"ID":"8def214d-3691-4efe-97fb-482d22899207","Type":"ContainerStarted","Data":"f9810b1edbf69df346a3b85d0b3804c8f8c09d11fdc4dc5dd83daceaa9dc5e9e"} Oct 30 02:23:36.473875 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:36.473821 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podStartSLOduration=1.473806683 podStartE2EDuration="1.473806683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:36.472462935 +0000 UTC m=+3244.864270551" watchObservedRunningTime="2026-10-30 02:23:36.473806683 +0000 UTC m=+3244.865614375" Oct 30 02:23:37.335373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.335307 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" podUID="4239de61-dff5-4d11-b232-6728936cc8b5" containerName="payload-processing" containerID="cri-o://3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea" gracePeriod=25 Oct 30 02:23:37.402933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.402894 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:37.409409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.409379 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:37.432156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.432098 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:37.433184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.433164 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:23:37.653878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.653848 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:23:37.721880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.721852 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume\") pod \"4239de61-dff5-4d11-b232-6728936cc8b5\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " Oct 30 02:23:37.722066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.721889 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume\") pod \"4239de61-dff5-4d11-b232-6728936cc8b5\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " Oct 30 02:23:37.722066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.721984 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials\") pod \"4239de61-dff5-4d11-b232-6728936cc8b5\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " Oct 30 02:23:37.722256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.722064 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q\") pod \"4239de61-dff5-4d11-b232-6728936cc8b5\" (UID: \"4239de61-dff5-4d11-b232-6728936cc8b5\") " Oct 30 02:23:37.722365 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.722339 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume" pod "4239de61-dff5-4d11-b232-6728936cc8b5" (UID: "4239de61-dff5-4d11-b232-6728936cc8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:37.722900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.722702 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume" pod "4239de61-dff5-4d11-b232-6728936cc8b5" (UID: "4239de61-dff5-4d11-b232-6728936cc8b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:37.722900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.722857 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:37.722900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.722877 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4239de61-dff5-4d11-b232-6728936cc8b5-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:37.755221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.739176 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials" pod "4239de61-dff5-4d11-b232-6728936cc8b5" (UID: "4239de61-dff5-4d11-b232-6728936cc8b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:37.755221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.739218 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q" pod "4239de61-dff5-4d11-b232-6728936cc8b5" (UID: "4239de61-dff5-4d11-b232-6728936cc8b5"). InnerVolumeSpecName "kube-api-access-w2z5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:37.824023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.823992 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:37.824023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:37.824018 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2z5q\" (UniqueName: \"kubernetes.io/projected/4239de61-dff5-4d11-b232-6728936cc8b5-kube-api-access-w2z5q\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:23:38.443202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.443167 6275 generic.go:356] "Generic (PLEG): container finished" podID="4239de61-dff5-4d11-b232-6728936cc8b5" containerID="3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea" exitCode=0 Oct 30 02:23:38.443709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.443217 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" Oct 30 02:23:38.443709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.443259 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" event={"ID":"4239de61-dff5-4d11-b232-6728936cc8b5","Type":"ContainerDied","Data":"3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea"} Oct 30 02:23:38.443709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.443318 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp" event={"ID":"4239de61-dff5-4d11-b232-6728936cc8b5","Type":"ContainerDied","Data":"e7a7bb0b5876a1f73524a8074cf2652e397538ef538b9bf7ff4a08353ffd063a"} Oct 30 02:23:38.443709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.443347 6275 scope.go:122] "RemoveContainer" containerID="3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea" Oct 30 02:23:38.456001 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.455984 6275 scope.go:122] "RemoveContainer" containerID="3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea" Oct 30 02:23:38.456336 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:23:38.456308 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea\": container with ID starting with 3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea not found: ID does not exist" containerID="3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea" Oct 30 02:23:38.456420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.456346 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea"} err="failed to get container status \"3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea\": rpc error: code = NotFound desc = could not find container \"3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea\": container with ID starting with 3580655552f49f908e2cfdd814880fa7f3627eeb6b64f7d766d34132b681e1ea not found: ID does not exist" Oct 30 02:23:38.491117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.491086 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp"] Oct 30 02:23:38.501167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:38.501144 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-574c95cfc5-p2vlp"] Oct 30 02:23:39.742979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:39.742944 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4239de61-dff5-4d11-b232-6728936cc8b5" path="/var/lib/kubelet/pods/4239de61-dff5-4d11-b232-6728936cc8b5/volumes" Oct 30 02:23:40.730478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.730444 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j"] Oct 30 02:23:40.731350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.731332 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4239de61-dff5-4d11-b232-6728936cc8b5" containerName="payload-processing" Oct 30 02:23:40.731350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.731350 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4239de61-dff5-4d11-b232-6728936cc8b5" containerName="payload-processing" Oct 30 02:23:40.731506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.731497 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4239de61-dff5-4d11-b232-6728936cc8b5" containerName="payload-processing" Oct 30 02:23:40.736888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.736858 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:40.751611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.751590 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3d5da6-dockercfg-hd47b\"" Oct 30 02:23:40.756587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.756566 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j"] Oct 30 02:23:40.762273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.762251 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d5da6\"" Oct 30 02:23:40.768788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.768760 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmrd\" (UniqueName: \"kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd\") pod \"payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:40.768914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.768808 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:40.903144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.899400 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmrd\" (UniqueName: \"kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd\") pod \"payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:40.903510 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.903479 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:40.933145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:40.914782 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:41.085241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:41.085204 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmrd\" (UniqueName: \"kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd\") pod \"payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:41.352942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:41.352862 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:41.676548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:41.676520 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j"] Oct 30 02:23:41.677754 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:23:41.677731 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0caf50_9763_416c_a137_f50803282abf.slice/crio-08afb31da745f8ecc7f34a0fd4896e07d6400cb181f9d13a4110a783d2d94499 WatchSource:0}: Error finding container 08afb31da745f8ecc7f34a0fd4896e07d6400cb181f9d13a4110a783d2d94499: Status 404 returned error can't find the container with id 08afb31da745f8ecc7f34a0fd4896e07d6400cb181f9d13a4110a783d2d94499 Oct 30 02:23:42.477190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:42.477152 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" event={"ID":"fa0caf50-9763-416c-a137-f50803282abf","Type":"ContainerStarted","Data":"127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9"} Oct 30 02:23:42.477627 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:42.477219 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" event={"ID":"fa0caf50-9763-416c-a137-f50803282abf","Type":"ContainerStarted","Data":"08afb31da745f8ecc7f34a0fd4896e07d6400cb181f9d13a4110a783d2d94499"} Oct 30 02:23:42.477627 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:42.477251 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:23:42.520551 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:42.520502 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" podStartSLOduration=2.520488774 podStartE2EDuration="2.520488774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:42.516679729 +0000 UTC m=+3250.908487367" watchObservedRunningTime="2026-10-30 02:23:42.520488774 +0000 UTC m=+3250.912296407" Oct 30 02:23:46.697807 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:23:46.697773 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:23:53.548822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:23:53.548787 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:24:16.699575 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:24:16.699535 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:24:32.455480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:32.455446 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:24:32.460440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:32.460417 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:24:46.700161 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:24:46.700109 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:24:51.471914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.471873 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9"] Oct 30 02:24:51.478235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.478213 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.494221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.494179 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"openshift-service-ca.crt\"" Oct 30 02:24:51.507396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.507366 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-95f40f\"" Oct 30 02:24:51.507539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.507432 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"agc-cred-43923dcebdf1\"" Oct 30 02:24:51.507539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.507434 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"kube-root-ca.crt\"" Oct 30 02:24:51.507539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.507484 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"payload-processing-external-model-e2e--6aaede7f-dockercfg-j4qbl\"" Oct 30 02:24:51.507751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.507597 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"routing-overlay\"" Oct 30 02:24:51.514361 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.514336 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9"] Oct 30 02:24:51.634617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.634582 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.634818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.634724 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.634876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.634854 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.634984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.634963 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77k4x\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.736656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.736570 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77k4x\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.736819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.736709 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.736819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.736809 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.736931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.736916 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.737642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.737613 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.737817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.737799 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.764193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.764158 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.767349 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.766952 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77k4x\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:51.793556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:51.793524 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:52.132930 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:52.132890 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9"] Oct 30 02:24:52.134509 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:24:52.134475 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747fb53b_9c56_4be6_9ae4_9f7bd7ea0b1c.slice/crio-6b42336d663b9505985518182c9cace4a8b1eb7708052b6ef523ff03f5ddd05c WatchSource:0}: Error finding container 6b42336d663b9505985518182c9cace4a8b1eb7708052b6ef523ff03f5ddd05c: Status 404 returned error can't find the container with id 6b42336d663b9505985518182c9cace4a8b1eb7708052b6ef523ff03f5ddd05c Oct 30 02:24:52.141188 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:52.140713 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9"] Oct 30 02:24:52.445214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:52.442080 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" event={"ID":"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c","Type":"ContainerStarted","Data":"2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619"} Oct 30 02:24:52.445214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:52.442170 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" event={"ID":"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c","Type":"ContainerStarted","Data":"6b42336d663b9505985518182c9cace4a8b1eb7708052b6ef523ff03f5ddd05c"} Oct 30 02:24:52.445214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:52.442373 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:53.475350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:53.475311 6275 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:24:53.475823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:53.475381 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:24:56.477849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.477747 6275 generic.go:356] "Generic (PLEG): container finished" podID="8def214d-3691-4efe-97fb-482d22899207" containerID="79c0b88f753f0038851b4f01a63ab9e72da4748ca7b6669189831639564e6ada" exitCode=0 Oct 30 02:24:56.477849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.477827 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" event={"ID":"8def214d-3691-4efe-97fb-482d22899207","Type":"ContainerDied","Data":"79c0b88f753f0038851b4f01a63ab9e72da4748ca7b6669189831639564e6ada"} Oct 30 02:24:56.478470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.478442 6275 scope.go:122] "RemoveContainer" containerID="79c0b88f753f0038851b4f01a63ab9e72da4748ca7b6669189831639564e6ada" Oct 30 02:24:56.480455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.480429 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:24:56.480547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.480495 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:24:56.480547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.480525 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:24:56.534518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:56.534469 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" podStartSLOduration=5.534454603 podStartE2EDuration="5.534454603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:24:52.489737716 +0000 UTC m=+3320.881545351" watchObservedRunningTime="2026-10-30 02:24:56.534454603 +0000 UTC m=+3324.926262236" Oct 30 02:24:57.451433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.451207 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" podUID="747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" containerName="payload-processing" containerID="cri-o://2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619" gracePeriod=25 Oct 30 02:24:57.488422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.488385 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" event={"ID":"8def214d-3691-4efe-97fb-482d22899207","Type":"ContainerStarted","Data":"5bcc2f90f28b52da11991962bb3eb34be0ad7f88ccc5a1cefbbef27d80c66f4a"} Oct 30 02:24:57.809753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.809727 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:57.872815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.872780 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume\") pod \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " Oct 30 02:24:57.873029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.872898 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials\") pod \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " Oct 30 02:24:57.873348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.873322 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x\") pod \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " Oct 30 02:24:57.873472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.873380 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume\") pod \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\" (UID: \"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c\") " Oct 30 02:24:57.874172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.873945 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume" pod "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" (UID: "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:24:57.874172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.874102 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:24:57.874172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.874137 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume" pod "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" (UID: "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:24:57.875808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.875772 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials" pod "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" (UID: "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:24:57.876423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.876399 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x" pod "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" (UID: "747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c"). InnerVolumeSpecName "kube-api-access-77k4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:24:57.975705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.975668 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77k4x\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-kube-api-access-77k4x\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:24:57.975705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.975705 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:24:57.975936 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:57.975718 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:24:58.408157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.407715 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:24:58.408157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.407850 6275 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 02:24:58.408157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.407890 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 02:24:58.497171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.497113 6275 generic.go:356] "Generic (PLEG): container finished" podID="747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" containerID="2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619" exitCode=0 Oct 30 02:24:58.497666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.497187 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" Oct 30 02:24:58.497666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.497216 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" event={"ID":"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c","Type":"ContainerDied","Data":"2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619"} Oct 30 02:24:58.497666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.497286 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9" event={"ID":"747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c","Type":"ContainerDied","Data":"6b42336d663b9505985518182c9cace4a8b1eb7708052b6ef523ff03f5ddd05c"} Oct 30 02:24:58.497666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.497320 6275 scope.go:122] "RemoveContainer" containerID="2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619" Oct 30 02:24:58.510814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.510795 6275 scope.go:122] "RemoveContainer" containerID="2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619" Oct 30 02:24:58.511079 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:24:58.511059 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619\": container with ID starting with 2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619 not found: ID does not exist" containerID="2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619" Oct 30 02:24:58.511153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.511089 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619"} err="failed to get container status \"2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619\": rpc error: code = NotFound desc = could not find container \"2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619\": container with ID starting with 2e7e7a3f41f2c43587e3d1e56ea9726c14dcdab76bda2b8f2d038335e2672619 not found: ID does not exist" Oct 30 02:24:58.566028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.565985 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9"] Oct 30 02:24:58.569818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:58.569793 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-bb85n24h9"] Oct 30 02:24:59.406702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:59.404417 6275 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 02:24:59.406702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:59.404517 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 02:24:59.766442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:24:59.766405 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" path="/var/lib/kubelet/pods/747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c/volumes" Oct 30 02:25:00.404539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:00.403656 6275 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 02:25:00.404539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:00.403748 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 02:25:01.403182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:01.403141 6275 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 02:25:01.403630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:01.403230 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 02:25:02.403851 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:02.403810 6275 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 02:25:02.404480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:02.403873 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 02:25:03.408381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:03.408350 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:25:03.408856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:03.408838 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:25:03.409449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:03.409430 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:25:12.433292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.432982 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2"] Oct 30 02:25:12.443572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.443542 6275 cpu_manager.go:414] "containerMap: removing container" podUID="747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" containerName="payload-processing" Oct 30 02:25:12.443572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.443571 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" containerName="payload-processing" Oct 30 02:25:12.443834 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.443799 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="747fb53b-9c56-4be6-9ae4-9f7bd7ea0b1c" containerName="payload-processing" Oct 30 02:25:12.448336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.448311 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2"] Oct 30 02:25:12.448522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.448501 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.469574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.469549 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-076a94\"/\"e2e-usage-sim-cdd1bf-cert\"" Oct 30 02:25:12.470409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.470384 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-076a94\"/\"kube-root-ca.crt\"" Oct 30 02:25:12.480043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.480020 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-076a94\"/\"openshift-service-ca.crt\"" Oct 30 02:25:12.480170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.480150 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-076a94\"/\"default-dockercfg-vh7ns\"" Oct 30 02:25:12.529722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.529687 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np25n\" (UniqueName: \"kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n\") pod \"e2e-usage-sim-cdd1bf-77d9d79986-ctfk2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.529897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.529868 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls\") pod \"e2e-usage-sim-cdd1bf-77d9d79986-ctfk2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.631274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.631238 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls\") pod \"e2e-usage-sim-cdd1bf-77d9d79986-ctfk2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.631453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.631431 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np25n\" (UniqueName: \"kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n\") pod \"e2e-usage-sim-cdd1bf-77d9d79986-ctfk2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.633770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.633746 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls\") pod \"e2e-usage-sim-cdd1bf-77d9d79986-ctfk2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.653684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.653654 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np25n\" (UniqueName: \"kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n\") pod \"e2e-usage-sim-cdd1bf-77d9d79986-ctfk2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:12.765705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:12.765676 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:13.109656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:13.109574 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2"] Oct 30 02:25:13.116392 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:25:13.116361 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdca2fbc_8f98_42b9_ae6b_61871ef803a2.slice/crio-9e9a18cbd57716aa46129e5b72e3d2b4a7568e29b25e27e4624b54ba5adbcc02 WatchSource:0}: Error finding container 9e9a18cbd57716aa46129e5b72e3d2b4a7568e29b25e27e4624b54ba5adbcc02: Status 404 returned error can't find the container with id 9e9a18cbd57716aa46129e5b72e3d2b4a7568e29b25e27e4624b54ba5adbcc02 Oct 30 02:25:13.741607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:13.741575 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" event={"ID":"fdca2fbc-8f98-42b9-ae6b-61871ef803a2","Type":"ContainerStarted","Data":"52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46"} Oct 30 02:25:13.742007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:13.741614 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" event={"ID":"fdca2fbc-8f98-42b9-ae6b-61871ef803a2","Type":"ContainerStarted","Data":"9e9a18cbd57716aa46129e5b72e3d2b4a7568e29b25e27e4624b54ba5adbcc02"} Oct 30 02:25:13.742007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:13.741634 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:13.742007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:13.741662 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:25:13.770297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:13.770247 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" podStartSLOduration=1.770234646 podStartE2EDuration="1.770234646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:13.769376712 +0000 UTC m=+3342.161184345" watchObservedRunningTime="2026-10-30 02:25:13.770234646 +0000 UTC m=+3342.162042278" Oct 30 02:25:16.410954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.410918 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t"] Oct 30 02:25:16.418298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.418275 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.444839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.444809 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2943ff\"" Oct 30 02:25:16.445003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.444881 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2943ff-dockercfg-4vnnq\"" Oct 30 02:25:16.450336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.450303 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t"] Oct 30 02:25:16.489462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.489425 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkvs\" (UniqueName: \"kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs\") pod \"payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.489660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.489596 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.619143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.600306 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkvs\" (UniqueName: \"kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs\") pod \"payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.619143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.600427 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.619143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.601546 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.645142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.639244 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x"] Oct 30 02:25:16.697197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.696406 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x"] Oct 30 02:25:16.697197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.696572 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:16.791386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.791267 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkvs\" (UniqueName: \"kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs\") pod \"payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:16.805471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.804824 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdtd\" (UniqueName: \"kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd\") pod \"payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:16.805471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.805107 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:16.834629 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:16.834595 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:25:16.935572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.935541 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdtd\" (UniqueName: \"kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd\") pod \"payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:16.936191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.936061 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:16.937251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.937219 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:16.957029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:16.956966 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdtd\" (UniqueName: \"kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd\") pod \"payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:17.037846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.037802 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:17.109900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.109597 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:17.367376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.367344 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t"] Oct 30 02:25:17.368998 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:25:17.368965 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca325f8_2e91_4ef7_bc44_1d412236c8cd.slice/crio-c0b56f2da38f25236273e3fce35e781d3befee927fb10ae041675a4920b396d0 WatchSource:0}: Error finding container c0b56f2da38f25236273e3fce35e781d3befee927fb10ae041675a4920b396d0: Status 404 returned error can't find the container with id c0b56f2da38f25236273e3fce35e781d3befee927fb10ae041675a4920b396d0 Oct 30 02:25:17.468846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.468812 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x"] Oct 30 02:25:17.470211 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:25:17.470183 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4105541b_32bc_4f12_b61c_2792bde77c6b.slice/crio-8b269be5a94b5423e33ae3f0b56a17047fbf64289e0cbd0cab04293dafacdb1f WatchSource:0}: Error finding container 8b269be5a94b5423e33ae3f0b56a17047fbf64289e0cbd0cab04293dafacdb1f: Status 404 returned error can't find the container with id 8b269be5a94b5423e33ae3f0b56a17047fbf64289e0cbd0cab04293dafacdb1f Oct 30 02:25:17.811947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.811907 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" event={"ID":"2ca325f8-2e91-4ef7-bc44-1d412236c8cd","Type":"ContainerStarted","Data":"1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7"} Oct 30 02:25:17.812294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.811977 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" event={"ID":"2ca325f8-2e91-4ef7-bc44-1d412236c8cd","Type":"ContainerStarted","Data":"c0b56f2da38f25236273e3fce35e781d3befee927fb10ae041675a4920b396d0"} Oct 30 02:25:17.812294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.812011 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:17.813824 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.813800 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" event={"ID":"4105541b-32bc-4f12-b61c-2792bde77c6b","Type":"ContainerStarted","Data":"0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3"} Oct 30 02:25:17.813941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.813848 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" event={"ID":"4105541b-32bc-4f12-b61c-2792bde77c6b","Type":"ContainerStarted","Data":"8b269be5a94b5423e33ae3f0b56a17047fbf64289e0cbd0cab04293dafacdb1f"} Oct 30 02:25:17.813941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.813879 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:17.881336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.881290 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" podStartSLOduration=1.8812771769999999 podStartE2EDuration="1.881277177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:17.881065989 +0000 UTC m=+3346.272873622" watchObservedRunningTime="2026-10-30 02:25:17.881277177 +0000 UTC m=+3346.273084801" Oct 30 02:25:17.929232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:17.929186 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" podStartSLOduration=1.929174334 podStartE2EDuration="1.929174334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:17.928213776 +0000 UTC m=+3346.320021424" watchObservedRunningTime="2026-10-30 02:25:17.929174334 +0000 UTC m=+3346.320981966" Oct 30 02:25:22.597674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.597635 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d"] Oct 30 02:25:22.602759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.602740 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.636007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.635458 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d"] Oct 30 02:25:22.636007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.635593 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d97e4c-dockercfg-w7flv\"" Oct 30 02:25:22.660185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.651608 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d97e4c\"" Oct 30 02:25:22.750141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.720233 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.754145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.750663 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt8g\" (UniqueName: \"kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g\") pod \"payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.881221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.878640 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.881221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.878884 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt8g\" (UniqueName: \"kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g\") pod \"payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.881221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.879740 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.908495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.908243 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt8g\" (UniqueName: \"kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g\") pod \"payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:22.927806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:22.926880 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:23.252951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:23.252923 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d"] Oct 30 02:25:23.255724 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:25:23.255603 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ab6efa_5a2a_4018_ba8f_5d64d52d9bbd.slice/crio-976deae7c35d282ce56524d3b4454f622c211a4f235d66a1f73ee0f2a3246136 WatchSource:0}: Error finding container 976deae7c35d282ce56524d3b4454f622c211a4f235d66a1f73ee0f2a3246136: Status 404 returned error can't find the container with id 976deae7c35d282ce56524d3b4454f622c211a4f235d66a1f73ee0f2a3246136 Oct 30 02:25:23.943820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:23.943778 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" event={"ID":"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd","Type":"ContainerStarted","Data":"36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f"} Oct 30 02:25:23.944247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:23.943850 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" event={"ID":"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd","Type":"ContainerStarted","Data":"976deae7c35d282ce56524d3b4454f622c211a4f235d66a1f73ee0f2a3246136"} Oct 30 02:25:23.944247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:23.944162 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:23.981024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:23.980975 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" podStartSLOduration=1.980963101 podStartE2EDuration="1.980963101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:23.980537253 +0000 UTC m=+3352.372344888" watchObservedRunningTime="2026-10-30 02:25:23.980963101 +0000 UTC m=+3352.372770734" Oct 30 02:25:25.914308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:25.914274 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l"] Oct 30 02:25:25.919524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:25.919500 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:25.971196 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:25.971167 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fddb7-openshift-default-dockercfg-rwsrf\"" Oct 30 02:25:26.090367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090320 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090398 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47cz\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090447 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090503 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090634 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090679 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090709 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090809 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.090928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.090899 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.129144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.115232 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l"] Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232090 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232185 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47cz\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232238 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232298 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232427 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232467 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232498 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232598 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.232679 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.233222 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.240193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.233502 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.273264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.271618 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.273264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.271668 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.273264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.271952 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.278147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.275637 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.278147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.277834 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47cz\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.282279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.282213 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.294388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.294326 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.538278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.538241 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:26.874076 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.874045 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l"] Oct 30 02:25:26.876072 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:25:26.876042 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4ccd06_2cc3_4bad_baae_7e0993e5ccf7.slice/crio-14ab3699a6ecc9b279707835da9d91ef9b98b158cd18954b60ef1f58bba9831a WatchSource:0}: Error finding container 14ab3699a6ecc9b279707835da9d91ef9b98b158cd18954b60ef1f58bba9831a: Status 404 returned error can't find the container with id 14ab3699a6ecc9b279707835da9d91ef9b98b158cd18954b60ef1f58bba9831a Oct 30 02:25:26.879117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.879081 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:25:26.879248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.879180 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:25:26.879248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.879216 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:25:26.969478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:26.969446 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" event={"ID":"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7","Type":"ContainerStarted","Data":"14ab3699a6ecc9b279707835da9d91ef9b98b158cd18954b60ef1f58bba9831a"} Oct 30 02:25:27.979029 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:27.978987 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" event={"ID":"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7","Type":"ContainerStarted","Data":"7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b"} Oct 30 02:25:28.043740 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.043680 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podStartSLOduration=3.043667092 podStartE2EDuration="3.043667092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:28.030824289 +0000 UTC m=+3356.422631921" watchObservedRunningTime="2026-10-30 02:25:28.043667092 +0000 UTC m=+3356.435474724" Oct 30 02:25:28.538664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.538635 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:28.544533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.544509 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:28.866541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.866466 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:25:28.866541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.866531 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:25:28.989924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.989884 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:28.991007 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:28.990987 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:25:32.707258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:32.707215 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9"] Oct 30 02:25:34.955224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:34.955188 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:25:35.415365 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.415336 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh"] Oct 30 02:25:35.421687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.421665 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.443865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.443827 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh"] Oct 30 02:25:35.548420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.548380 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.548599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.548560 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.548726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.548704 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8mv\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.548874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.548757 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.650818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.650729 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.650973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.650842 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8mv\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.650973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.650871 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.651053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.650981 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.651579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.651552 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.651700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.651562 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.653507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.653483 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.673776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.673753 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8mv\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:35.748787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:35.748681 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:36.472415 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:36.472382 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh"] Oct 30 02:25:36.474258 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:25:36.474229 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b98739_b03f_4b75_b224_3f184e7625c9.slice/crio-449bccfb36bc17f70aaea086bcc3ea704a8a7f60ef4d429efe3c49cc7edd63bc WatchSource:0}: Error finding container 449bccfb36bc17f70aaea086bcc3ea704a8a7f60ef4d429efe3c49cc7edd63bc: Status 404 returned error can't find the container with id 449bccfb36bc17f70aaea086bcc3ea704a8a7f60ef4d429efe3c49cc7edd63bc Oct 30 02:25:37.018351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.018314 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh"] Oct 30 02:25:37.030257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.030224 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf"] Oct 30 02:25:37.101426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.101389 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" event={"ID":"30b98739-b03f-4b75-b224-3f184e7625c9","Type":"ContainerStarted","Data":"94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256"} Oct 30 02:25:37.101623 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.101484 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" event={"ID":"30b98739-b03f-4b75-b224-3f184e7625c9","Type":"ContainerStarted","Data":"449bccfb36bc17f70aaea086bcc3ea704a8a7f60ef4d429efe3c49cc7edd63bc"} Oct 30 02:25:37.101623 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.101613 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:37.101723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.101620 6275 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" secret="" err="secret \"payload-processing-external-model-e2e--4b9246f1-dockercfg-2dswk\" not found" Oct 30 02:25:37.134362 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.134328 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-58fad4/agc-cred-4777b568e245: secret "agc-cred-4777b568e245" not found Oct 30 02:25:37.134362 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.134358 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh: secret "agc-cred-4777b568e245" not found Oct 30 02:25:37.134605 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.134417 6275 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-plugins-e2e-extmod-a-58fad4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:37.134605 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.134423 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:37.634401227 +0000 UTC m=+3366.026208853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : secret "agc-cred-4777b568e245" not found Oct 30 02:25:37.134605 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.134503 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:37.634487041 +0000 UTC m=+3366.026294662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:37.142359 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.142309 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" podStartSLOduration=2.14229485 podStartE2EDuration="2.14229485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:37.141830967 +0000 UTC m=+3365.533638614" watchObservedRunningTime="2026-10-30 02:25:37.14229485 +0000 UTC m=+3365.534102482" Oct 30 02:25:37.652198 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.652161 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-58fad4/agc-cred-4777b568e245: secret "agc-cred-4777b568e245" not found Oct 30 02:25:37.652198 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.652190 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh: secret "agc-cred-4777b568e245" not found Oct 30 02:25:37.652198 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.652196 6275 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-plugins-e2e-extmod-a-58fad4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:37.652669 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.652266 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:38.652250763 +0000 UTC m=+3367.044058373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:37.652669 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:37.652282 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:38.652275888 +0000 UTC m=+3367.044083500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : secret "agc-cred-4777b568e245" not found Oct 30 02:25:37.707535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:37.707475 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" podUID="15351072-7614-4611-9de9-bcaabf3310ab" containerName="payload-processing" containerID="cri-o://b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6" gracePeriod=25 Oct 30 02:25:38.014651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.014621 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:25:38.059074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.058411 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume\") pod \"15351072-7614-4611-9de9-bcaabf3310ab\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " Oct 30 02:25:38.059074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.058588 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp\") pod \"15351072-7614-4611-9de9-bcaabf3310ab\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " Oct 30 02:25:38.059074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.058632 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials\") pod \"15351072-7614-4611-9de9-bcaabf3310ab\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " Oct 30 02:25:38.059074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.058825 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume\") pod \"15351072-7614-4611-9de9-bcaabf3310ab\" (UID: \"15351072-7614-4611-9de9-bcaabf3310ab\") " Oct 30 02:25:38.060475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.059958 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume" pod "15351072-7614-4611-9de9-bcaabf3310ab" (UID: "15351072-7614-4611-9de9-bcaabf3310ab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:38.061096 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.061068 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume" pod "15351072-7614-4611-9de9-bcaabf3310ab" (UID: "15351072-7614-4611-9de9-bcaabf3310ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:38.062819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.062795 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp" pod "15351072-7614-4611-9de9-bcaabf3310ab" (UID: "15351072-7614-4611-9de9-bcaabf3310ab"). InnerVolumeSpecName "kube-api-access-s4lsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:38.062951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.062895 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials" pod "15351072-7614-4611-9de9-bcaabf3310ab" (UID: "15351072-7614-4611-9de9-bcaabf3310ab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:38.110249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.110210 6275 generic.go:356] "Generic (PLEG): container finished" podID="15351072-7614-4611-9de9-bcaabf3310ab" containerID="b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6" exitCode=0 Oct 30 02:25:38.110423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.110255 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" Oct 30 02:25:38.110423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.110304 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" event={"ID":"15351072-7614-4611-9de9-bcaabf3310ab","Type":"ContainerDied","Data":"b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6"} Oct 30 02:25:38.110423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.110355 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9" event={"ID":"15351072-7614-4611-9de9-bcaabf3310ab","Type":"ContainerDied","Data":"714858346fa8d7b6090c06d07c7bfacbc146c292fe1c31fee4eb8bbd7036f3f5"} Oct 30 02:25:38.110423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.110382 6275 scope.go:122] "RemoveContainer" containerID="b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6" Oct 30 02:25:38.126038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.126019 6275 scope.go:122] "RemoveContainer" containerID="b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6" Oct 30 02:25:38.126308 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:38.126289 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6\": container with ID starting with b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6 not found: ID does not exist" containerID="b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6" Oct 30 02:25:38.126378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.126317 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6"} err="failed to get container status \"b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6\": rpc error: code = NotFound desc = could not find container \"b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6\": container with ID starting with b0d41e93a1d81ea3f1e593cea04c6181f2fb7963aa7d2ed48834668798a221f6 not found: ID does not exist" Oct 30 02:25:38.160864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.160833 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.160864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.160862 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15351072-7614-4611-9de9-bcaabf3310ab-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.161056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.160872 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4lsp\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-kube-api-access-s4lsp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.161056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.160880 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15351072-7614-4611-9de9-bcaabf3310ab-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.204088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.203614 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9"] Oct 30 02:25:38.210924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:38.210873 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4dgcqz9"] Oct 30 02:25:38.680998 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:38.680965 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-58fad4/agc-cred-4777b568e245: secret "agc-cred-4777b568e245" not found Oct 30 02:25:38.680998 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:38.680992 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh: secret "agc-cred-4777b568e245" not found Oct 30 02:25:38.680998 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:38.680997 6275 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-plugins-e2e-extmod-a-58fad4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:38.681605 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:38.681054 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:40.681036598 +0000 UTC m=+3369.072844209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : secret "agc-cred-4777b568e245" not found Oct 30 02:25:38.681605 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:38.681074 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:40.681065274 +0000 UTC m=+3369.072872885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:39.741987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:39.741959 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15351072-7614-4611-9de9-bcaabf3310ab" path="/var/lib/kubelet/pods/15351072-7614-4611-9de9-bcaabf3310ab/volumes" Oct 30 02:25:40.712865 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:40.712823 6275 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-58fad4/agc-cred-4777b568e245: secret "agc-cred-4777b568e245" not found Oct 30 02:25:40.712865 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:40.712848 6275 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-plugins-e2e-extmod-a-58fad4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:40.713113 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:40.712918 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:44.712900954 +0000 UTC m=+3373.104708582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-58fad4" not found Oct 30 02:25:40.713113 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:40.712854 6275 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh: secret "agc-cred-4777b568e245" not found Oct 30 02:25:40.713113 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:40.713005 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials podName:30b98739-b03f-4b75-b224-3f184e7625c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:25:44.712988903 +0000 UTC m=+3373.104796533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9") : secret "agc-cred-4777b568e245" not found Oct 30 02:25:42.030692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.030644 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" podUID="0d432e06-0395-4d92-9758-4d76d190fbdc" containerName="payload-processing" containerID="cri-o://d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a" gracePeriod=25 Oct 30 02:25:42.316446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.316413 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:25:42.443995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.443954 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials\") pod \"0d432e06-0395-4d92-9758-4d76d190fbdc\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " Oct 30 02:25:42.444204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444084 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg\") pod \"0d432e06-0395-4d92-9758-4d76d190fbdc\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " Oct 30 02:25:42.444204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444151 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume\") pod \"0d432e06-0395-4d92-9758-4d76d190fbdc\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " Oct 30 02:25:42.444354 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444277 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume\") pod \"0d432e06-0395-4d92-9758-4d76d190fbdc\" (UID: \"0d432e06-0395-4d92-9758-4d76d190fbdc\") " Oct 30 02:25:42.444573 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444549 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume" pod "0d432e06-0395-4d92-9758-4d76d190fbdc" (UID: "0d432e06-0395-4d92-9758-4d76d190fbdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:42.444702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444612 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume" pod "0d432e06-0395-4d92-9758-4d76d190fbdc" (UID: "0d432e06-0395-4d92-9758-4d76d190fbdc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:42.444990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444969 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:42.444990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.444990 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d432e06-0395-4d92-9758-4d76d190fbdc-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:42.446090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.446068 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials" pod "0d432e06-0395-4d92-9758-4d76d190fbdc" (UID: "0d432e06-0395-4d92-9758-4d76d190fbdc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:42.446322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.446301 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg" pod "0d432e06-0395-4d92-9758-4d76d190fbdc" (UID: "0d432e06-0395-4d92-9758-4d76d190fbdc"). InnerVolumeSpecName "kube-api-access-whspg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:42.546842 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.546743 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:42.546842 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:42.546787 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whspg\" (UniqueName: \"kubernetes.io/projected/0d432e06-0395-4d92-9758-4d76d190fbdc-kube-api-access-whspg\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.110917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.110875 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" podUID="30b98739-b03f-4b75-b224-3f184e7625c9" containerName="payload-processing" containerID="cri-o://94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256" gracePeriod=25 Oct 30 02:25:43.152856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.152824 6275 generic.go:356] "Generic (PLEG): container finished" podID="0d432e06-0395-4d92-9758-4d76d190fbdc" containerID="d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a" exitCode=0 Oct 30 02:25:43.153003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.152872 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" Oct 30 02:25:43.153003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.152914 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" event={"ID":"0d432e06-0395-4d92-9758-4d76d190fbdc","Type":"ContainerDied","Data":"d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a"} Oct 30 02:25:43.153003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.152978 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf" event={"ID":"0d432e06-0395-4d92-9758-4d76d190fbdc","Type":"ContainerDied","Data":"3bb6c37a28051e2be399fdd9eaf55de671046fb9f29641aeeb6e389918900e9f"} Oct 30 02:25:43.153204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.153008 6275 scope.go:122] "RemoveContainer" containerID="d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a" Oct 30 02:25:43.223315 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.223284 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf"] Oct 30 02:25:43.225891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.225868 6275 scope.go:122] "RemoveContainer" containerID="d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a" Oct 30 02:25:43.226227 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:43.226207 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a\": container with ID starting with d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a not found: ID does not exist" containerID="d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a" Oct 30 02:25:43.226292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.226237 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a"} err="failed to get container status \"d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a\": rpc error: code = NotFound desc = could not find container \"d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a\": container with ID starting with d46ae043c58c049c243f86e158ff7bfe6589dc1ddf48f81cce1e6379940f338a not found: ID does not exist" Oct 30 02:25:43.232558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.232532 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7864b9jrf"] Oct 30 02:25:43.474322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.474291 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:43.587518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.587488 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume\") pod \"30b98739-b03f-4b75-b224-3f184e7625c9\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " Oct 30 02:25:43.587749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.587674 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv\") pod \"30b98739-b03f-4b75-b224-3f184e7625c9\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " Oct 30 02:25:43.587749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.587718 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials\") pod \"30b98739-b03f-4b75-b224-3f184e7625c9\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " Oct 30 02:25:43.587991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.587827 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume\") pod \"30b98739-b03f-4b75-b224-3f184e7625c9\" (UID: \"30b98739-b03f-4b75-b224-3f184e7625c9\") " Oct 30 02:25:43.588175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.588038 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume" pod "30b98739-b03f-4b75-b224-3f184e7625c9" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:43.588270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.588213 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume" pod "30b98739-b03f-4b75-b224-3f184e7625c9" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:43.588574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.588553 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.588657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.588576 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b98739-b03f-4b75-b224-3f184e7625c9-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.589960 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.589936 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials" pod "30b98739-b03f-4b75-b224-3f184e7625c9" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:43.590068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.589990 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv" pod "30b98739-b03f-4b75-b224-3f184e7625c9" (UID: "30b98739-b03f-4b75-b224-3f184e7625c9"). InnerVolumeSpecName "kube-api-access-sg8mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:43.689741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.689700 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.689741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.689739 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8mv\" (UniqueName: \"kubernetes.io/projected/30b98739-b03f-4b75-b224-3f184e7625c9-kube-api-access-sg8mv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.741692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:43.741620 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d432e06-0395-4d92-9758-4d76d190fbdc" path="/var/lib/kubelet/pods/0d432e06-0395-4d92-9758-4d76d190fbdc/volumes" Oct 30 02:25:44.163227 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.163192 6275 generic.go:356] "Generic (PLEG): container finished" podID="30b98739-b03f-4b75-b224-3f184e7625c9" containerID="94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256" exitCode=0 Oct 30 02:25:44.163739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.163242 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" Oct 30 02:25:44.163739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.163272 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" event={"ID":"30b98739-b03f-4b75-b224-3f184e7625c9","Type":"ContainerDied","Data":"94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256"} Oct 30 02:25:44.163739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.163323 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh" event={"ID":"30b98739-b03f-4b75-b224-3f184e7625c9","Type":"ContainerDied","Data":"449bccfb36bc17f70aaea086bcc3ea704a8a7f60ef4d429efe3c49cc7edd63bc"} Oct 30 02:25:44.163739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.163347 6275 scope.go:122] "RemoveContainer" containerID="94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256" Oct 30 02:25:44.174341 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.174320 6275 scope.go:122] "RemoveContainer" containerID="94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256" Oct 30 02:25:44.174624 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:44.174601 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256\": container with ID starting with 94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256 not found: ID does not exist" containerID="94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256" Oct 30 02:25:44.174701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.174631 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256"} err="failed to get container status \"94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256\": rpc error: code = NotFound desc = could not find container \"94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256\": container with ID starting with 94fc3a6b9227df7b4116c7f6d2ea5c3923471c3f26bc7e839f29da88cb25e256 not found: ID does not exist" Oct 30 02:25:44.211599 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.211572 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh"] Oct 30 02:25:44.221616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:44.221593 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-7bcdksfzh"] Oct 30 02:25:45.810418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:45.810377 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b98739-b03f-4b75-b224-3f184e7625c9" path="/var/lib/kubelet/pods/30b98739-b03f-4b75-b224-3f184e7625c9/volumes" Oct 30 02:25:46.741412 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:25:46.741362 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:25:51.247171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.247093 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p"] Oct 30 02:25:51.248080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248061 6275 cpu_manager.go:414] "containerMap: removing container" podUID="15351072-7614-4611-9de9-bcaabf3310ab" containerName="payload-processing" Oct 30 02:25:51.248174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248080 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15351072-7614-4611-9de9-bcaabf3310ab" containerName="payload-processing" Oct 30 02:25:51.248174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248097 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0d432e06-0395-4d92-9758-4d76d190fbdc" containerName="payload-processing" Oct 30 02:25:51.248174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248103 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d432e06-0395-4d92-9758-4d76d190fbdc" containerName="payload-processing" Oct 30 02:25:51.248174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248110 6275 cpu_manager.go:414] "containerMap: removing container" podUID="30b98739-b03f-4b75-b224-3f184e7625c9" containerName="payload-processing" Oct 30 02:25:51.248174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248114 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b98739-b03f-4b75-b224-3f184e7625c9" containerName="payload-processing" Oct 30 02:25:51.248356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248328 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="15351072-7614-4611-9de9-bcaabf3310ab" containerName="payload-processing" Oct 30 02:25:51.248356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248340 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d432e06-0395-4d92-9758-4d76d190fbdc" containerName="payload-processing" Oct 30 02:25:51.248356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.248346 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b98739-b03f-4b75-b224-3f184e7625c9" containerName="payload-processing" Oct 30 02:25:51.253754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.253734 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.274191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.274163 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p"] Oct 30 02:25:51.278496 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.278473 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-db01c4-dockercfg-8tc67\"" Oct 30 02:25:51.278951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.278932 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-db01c4\"" Oct 30 02:25:51.326581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.326544 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.326961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.326936 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tcm\" (UniqueName: \"kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm\") pod \"payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.498450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.429100 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.498450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.429240 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tcm\" (UniqueName: \"kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm\") pod \"payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.498450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.430766 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.632064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.627844 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6"] Oct 30 02:25:51.644657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.644626 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.649854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.649821 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6"] Oct 30 02:25:51.658877 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.658842 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tcm\" (UniqueName: \"kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm\") pod \"payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.840137 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.840077 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.840137 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.840138 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mft\" (UniqueName: \"kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft\") pod \"payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.872539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.872503 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:51.944041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.943000 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.944041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.943056 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mft\" (UniqueName: \"kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft\") pod \"payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.944041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.943990 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.981146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.980726 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mft\" (UniqueName: \"kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft\") pod \"payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:51.991310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:51.990184 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:52.385540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:52.361607 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p"] Oct 30 02:25:52.737720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:52.737672 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6"] Oct 30 02:25:53.427767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.427729 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" event={"ID":"29f620af-6b5a-43d9-b385-4160459c791f","Type":"ContainerStarted","Data":"c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f"} Oct 30 02:25:53.428394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.428369 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" event={"ID":"29f620af-6b5a-43d9-b385-4160459c791f","Type":"ContainerStarted","Data":"f287e748093bfc6f1aef8a1d1f395766dc97a923d85d227f2bca2883eb112689"} Oct 30 02:25:53.428503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.428450 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:25:53.429682 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.429656 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" event={"ID":"5ca45faa-a462-479f-9a48-9ab128512db4","Type":"ContainerStarted","Data":"944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5"} Oct 30 02:25:53.429797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.429699 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" event={"ID":"5ca45faa-a462-479f-9a48-9ab128512db4","Type":"ContainerStarted","Data":"e4558dc3918c036bba903755dc6efbbbf7ecd75daf7d3c56e5f4c43a63277b5c"} Oct 30 02:25:53.429946 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.429928 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:25:53.514143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.512456 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" podStartSLOduration=2.51243418 podStartE2EDuration="2.51243418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:53.474414876 +0000 UTC m=+3381.866222511" watchObservedRunningTime="2026-10-30 02:25:53.51243418 +0000 UTC m=+3381.904241814" Oct 30 02:25:53.524170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:25:53.520632 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" podStartSLOduration=2.520608899 podStartE2EDuration="2.520608899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:53.512573772 +0000 UTC m=+3381.904381407" watchObservedRunningTime="2026-10-30 02:25:53.520608899 +0000 UTC m=+3381.912416532" Oct 30 02:26:03.920221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:03.920185 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k"] Oct 30 02:26:03.925133 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:03.925100 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:03.941976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:03.941951 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2943ff-serving-cert\"" Oct 30 02:26:03.956757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:03.956731 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k"] Oct 30 02:26:04.060602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.060558 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.060786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.060646 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5tl\" (UniqueName: \"kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.060786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.060753 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.060906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.060891 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.190191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.182335 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.190191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.182581 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.190191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.182626 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5tl\" (UniqueName: \"kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.190191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.182777 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.190191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.184331 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.195809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.195774 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.206409 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.206376 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.266146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.257924 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5tl\" (UniqueName: \"kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl\") pod \"maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.448534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.448449 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:26:04.539799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.539764 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:04.891604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.891561 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k"] Oct 30 02:26:04.902568 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:26:04.902516 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46345244_b781_451e_a9d1_5af8558860d8.slice/crio-e7c3e2e32ae026f344e4b74d181843ea10913fe2ec4e46b22abac33aa89e4ac2 WatchSource:0}: Error finding container e7c3e2e32ae026f344e4b74d181843ea10913fe2ec4e46b22abac33aa89e4ac2: Status 404 returned error can't find the container with id e7c3e2e32ae026f344e4b74d181843ea10913fe2ec4e46b22abac33aa89e4ac2 Oct 30 02:26:04.904465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:04.904437 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:26:05.454906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:05.454875 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:26:05.583842 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:05.583808 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" event={"ID":"46345244-b781-451e-a9d1-5af8558860d8","Type":"ContainerStarted","Data":"e7c3e2e32ae026f344e4b74d181843ea10913fe2ec4e46b22abac33aa89e4ac2"} Oct 30 02:26:06.593856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:06.593815 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" event={"ID":"46345244-b781-451e-a9d1-5af8558860d8","Type":"ContainerStarted","Data":"caf76922d9daba1f1bb60acdd74a4d38f663bcf9a6183feb5583a53cb4229022"} Oct 30 02:26:06.594436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:06.593964 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:06.634376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:06.634324 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" podStartSLOduration=3.071532239 podStartE2EDuration="3.634306769s" totalImagesPullingTime="562.77453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:26:06.633083648 +0000 UTC m=+3395.024891281" watchObservedRunningTime="2026-10-30 02:26:06.634306769 +0000 UTC m=+3395.026114419" Oct 30 02:26:12.607099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:12.607070 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:26:16.698343 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:26:16.698301 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:26:39.377042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.377005 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns"] Oct 30 02:26:39.383280 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.383260 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.405337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.405307 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-db01c4-serving-cert\"" Oct 30 02:26:39.411778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.411748 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns"] Oct 30 02:26:39.529665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.529630 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtvb\" (UniqueName: \"kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.529850 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.529678 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.529895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.529856 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.529975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.529963 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.651222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.647254 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtvb\" (UniqueName: \"kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.651222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.647778 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.651222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.648173 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.651222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.648320 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.651222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.650000 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.683784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.682815 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.687820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.686819 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtvb\" (UniqueName: \"kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.687820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.687221 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs\") pod \"maas-api-e2e-inf-a-db01c4-6484556949-kf4ns\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:39.699329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:39.699038 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:40.023477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:40.023448 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns"] Oct 30 02:26:40.025867 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:26:40.025842 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61926654_587a_4e1c_b209_d0ef76657bca.slice/crio-d98ae14e2f9ee5a34edc302bf2ed580f2b3c33af5b413bdc8df5bcd387bd86b2 WatchSource:0}: Error finding container d98ae14e2f9ee5a34edc302bf2ed580f2b3c33af5b413bdc8df5bcd387bd86b2: Status 404 returned error can't find the container with id d98ae14e2f9ee5a34edc302bf2ed580f2b3c33af5b413bdc8df5bcd387bd86b2 Oct 30 02:26:41.013925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:41.013888 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" event={"ID":"61926654-587a-4e1c-b209-d0ef76657bca","Type":"ContainerStarted","Data":"8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756"} Oct 30 02:26:41.014450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:41.013940 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" event={"ID":"61926654-587a-4e1c-b209-d0ef76657bca","Type":"ContainerStarted","Data":"d98ae14e2f9ee5a34edc302bf2ed580f2b3c33af5b413bdc8df5bcd387bd86b2"} Oct 30 02:26:41.014450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:41.013963 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:26:41.055477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:41.055419 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" podStartSLOduration=1.712120436 podStartE2EDuration="2.055403624s" totalImagesPullingTime="343.283188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:26:41.052805732 +0000 UTC m=+3429.444613395" watchObservedRunningTime="2026-10-30 02:26:41.055403624 +0000 UTC m=+3429.447211256" Oct 30 02:26:46.699309 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:26:46.699224 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:26:47.028048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:26:47.028017 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:27:08.424837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.424796 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp"] Oct 30 02:27:08.452187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.452154 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.516160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.514832 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4593af\"" Oct 30 02:27:08.516160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.514905 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4593af-dockercfg-x6dzw\"" Oct 30 02:27:08.572874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.572391 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp"] Oct 30 02:27:08.664201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.656377 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.667064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.664583 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9qs\" (UniqueName: \"kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs\") pod \"payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.765961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.765921 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.766228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.766039 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9qs\" (UniqueName: \"kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs\") pod \"payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.767066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.767042 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.788543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.788510 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9qs\" (UniqueName: \"kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs\") pod \"payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:08.896874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:08.896835 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:09.216300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:09.216266 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp"] Oct 30 02:27:09.217688 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:27:09.217661 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5bde7e_c056_43f6_809c_623ea8d060a8.slice/crio-dc865c7f42c74454d612a587d017d3a5f087bf3fa6aab6c309fec3a4025b007e WatchSource:0}: Error finding container dc865c7f42c74454d612a587d017d3a5f087bf3fa6aab6c309fec3a4025b007e: Status 404 returned error can't find the container with id dc865c7f42c74454d612a587d017d3a5f087bf3fa6aab6c309fec3a4025b007e Oct 30 02:27:09.408027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:09.407973 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" event={"ID":"ee5bde7e-c056-43f6-809c-623ea8d060a8","Type":"ContainerStarted","Data":"dc865c7f42c74454d612a587d017d3a5f087bf3fa6aab6c309fec3a4025b007e"} Oct 30 02:27:10.455845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:10.455802 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" event={"ID":"ee5bde7e-c056-43f6-809c-623ea8d060a8","Type":"ContainerStarted","Data":"2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc"} Oct 30 02:27:10.469584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:10.469555 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:16.699100 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:16.699055 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:27:21.479600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:21.479547 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:27:21.518159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:21.518090 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" podStartSLOduration=13.51807682 podStartE2EDuration="13.51807682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:10.538403645 +0000 UTC m=+3458.930211289" watchObservedRunningTime="2026-10-30 02:27:21.51807682 +0000 UTC m=+3469.909884453" Oct 30 02:27:22.592692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:22.592658 6275 generic.go:356] "Generic (PLEG): container finished" podID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerID="7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b" exitCode=0 Oct 30 02:27:22.592692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:22.592697 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" event={"ID":"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7","Type":"ContainerDied","Data":"7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b"} Oct 30 02:27:22.593188 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:22.593171 6275 scope.go:122] "RemoveContainer" containerID="7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b" Oct 30 02:27:22.595262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:22.595227 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:27:22.595391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:22.595316 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:27:22.595391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:22.595366 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:27:23.601947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:23.601915 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" event={"ID":"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7","Type":"ContainerStarted","Data":"35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981"} Oct 30 02:27:24.561936 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.561901 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:27:24.566101 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.566068 6275 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:24.566469 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.566436 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 02:27:24.723979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.723942 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw"] Oct 30 02:27:24.736064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.736032 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:24.781393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.773413 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c75203\"" Oct 30 02:27:24.781393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.773546 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c75203-dockercfg-wbj6l\"" Oct 30 02:27:24.832687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.831988 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgpx\" (UniqueName: \"kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx\") pod \"payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:24.832687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.832352 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:24.854264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.854220 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw"] Oct 30 02:27:24.956605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.956561 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgpx\" (UniqueName: \"kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx\") pod \"payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:24.985469 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.985427 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:24.987350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:24.987320 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:25.014386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:25.014353 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgpx\" (UniqueName: \"kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx\") pod \"payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:25.174110 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:25.174025 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:25.512464 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:27:25.512421 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb24450c_9ff5_45b0_98d8_721ae6b05aae.slice/crio-5d2490fec434e1f6451affc9945824a7adbaa767ae07d441607472ec5dbb09fe WatchSource:0}: Error finding container 5d2490fec434e1f6451affc9945824a7adbaa767ae07d441607472ec5dbb09fe: Status 404 returned error can't find the container with id 5d2490fec434e1f6451affc9945824a7adbaa767ae07d441607472ec5dbb09fe Oct 30 02:27:25.513904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:25.513877 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw"] Oct 30 02:27:25.554154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:25.553645 6275 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:25.554154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:25.553709 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 02:27:25.629727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:25.629698 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" event={"ID":"bb24450c-9ff5-45b0-98d8-721ae6b05aae","Type":"ContainerStarted","Data":"5d2490fec434e1f6451affc9945824a7adbaa767ae07d441607472ec5dbb09fe"} Oct 30 02:27:26.551561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:26.551525 6275 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:26.552211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:26.552176 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 02:27:26.672034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:26.671994 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" event={"ID":"bb24450c-9ff5-45b0-98d8-721ae6b05aae","Type":"ContainerStarted","Data":"2468873a0117ed9ef17ce3d05b508736602bb3eafc4d7231cbfd507bd3bffcbf"} Oct 30 02:27:26.672312 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:26.672294 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:26.713557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:26.713492 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" podStartSLOduration=2.713478367 podStartE2EDuration="2.713478367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:26.709767889 +0000 UTC m=+3475.101575523" watchObservedRunningTime="2026-10-30 02:27:26.713478367 +0000 UTC m=+3475.105285999" Oct 30 02:27:27.546604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:27.546568 6275 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:27.546907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:27.546874 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 02:27:28.542567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:28.542526 6275 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:28.543221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:28.543189 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 02:27:28.990381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:28.990285 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:27:29.544100 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:29.544068 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:27:29.697905 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:29.697875 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:27:37.683319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:37.683288 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:27:38.023022 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:38.022990 6275 secret.go:189] Couldn't get secret e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-cert: secret "e2e-usage-sim-cdd1bf-cert" not found Oct 30 02:27:38.023420 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:38.023392 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls podName:fdca2fbc-8f98-42b9-ae6b-61871ef803a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:27:38.523367486 +0000 UTC m=+3486.915175100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls") pod "e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" (UID: "fdca2fbc-8f98-42b9-ae6b-61871ef803a2") : secret "e2e-usage-sim-cdd1bf-cert" not found Oct 30 02:27:38.031310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.029939 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2"] Oct 30 02:27:38.031310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.030069 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" podUID="fdca2fbc-8f98-42b9-ae6b-61871ef803a2" containerName="sim" containerID="cri-o://52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46" gracePeriod=30 Oct 30 02:27:38.552152 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:38.552108 6275 secret.go:189] Couldn't get secret e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-cert: secret "e2e-usage-sim-cdd1bf-cert" not found Oct 30 02:27:38.552324 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:38.552192 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls podName:fdca2fbc-8f98-42b9-ae6b-61871ef803a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:27:39.55217735 +0000 UTC m=+3487.943984977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls") pod "e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" (UID: "fdca2fbc-8f98-42b9-ae6b-61871ef803a2") : secret "e2e-usage-sim-cdd1bf-cert" not found Oct 30 02:27:38.566009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.565981 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:27:38.658114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.653975 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls\" (UniqueName: \"kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls\") pod \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " Oct 30 02:27:38.658114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.654056 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n\" (UniqueName: \"kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n\") pod \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\" (UID: \"fdca2fbc-8f98-42b9-ae6b-61871ef803a2\") " Oct 30 02:27:38.661874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.661840 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n" pod "fdca2fbc-8f98-42b9-ae6b-61871ef803a2" (UID: "fdca2fbc-8f98-42b9-ae6b-61871ef803a2"). InnerVolumeSpecName "kube-api-access-np25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:38.666180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.665585 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls" pod "fdca2fbc-8f98-42b9-ae6b-61871ef803a2" (UID: "fdca2fbc-8f98-42b9-ae6b-61871ef803a2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:27:38.771147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.770156 6275 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:27:38.771147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:38.770207 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np25n\" (UniqueName: \"kubernetes.io/projected/fdca2fbc-8f98-42b9-ae6b-61871ef803a2-kube-api-access-np25n\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:27:39.043974 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.043931 6275 generic.go:356] "Generic (PLEG): container finished" podID="fdca2fbc-8f98-42b9-ae6b-61871ef803a2" containerID="52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46" exitCode=0 Oct 30 02:27:39.044204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.044045 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" event={"ID":"fdca2fbc-8f98-42b9-ae6b-61871ef803a2","Type":"ContainerDied","Data":"52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46"} Oct 30 02:27:39.044204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.044084 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" event={"ID":"fdca2fbc-8f98-42b9-ae6b-61871ef803a2","Type":"ContainerDied","Data":"9e9a18cbd57716aa46129e5b72e3d2b4a7568e29b25e27e4624b54ba5adbcc02"} Oct 30 02:27:39.044204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.044099 6275 scope.go:122] "RemoveContainer" containerID="52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46" Oct 30 02:27:39.045233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.045207 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2" Oct 30 02:27:39.059581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.059564 6275 scope.go:122] "RemoveContainer" containerID="52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46" Oct 30 02:27:39.059936 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:39.059885 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46\": container with ID starting with 52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46 not found: ID does not exist" containerID="52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46" Oct 30 02:27:39.059936 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.059921 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46"} err="failed to get container status \"52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46\": rpc error: code = NotFound desc = could not find container \"52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46\": container with ID starting with 52489da1b31925a7b45ea520b84b5adfb066552152fc0aad187ce5c9bc032b46 not found: ID does not exist" Oct 30 02:27:39.093480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.093446 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2"] Oct 30 02:27:39.103464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.103437 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-076a94/e2e-usage-sim-cdd1bf-77d9d79986-ctfk2"] Oct 30 02:27:39.743589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:39.743550 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdca2fbc-8f98-42b9-ae6b-61871ef803a2" path="/var/lib/kubelet/pods/fdca2fbc-8f98-42b9-ae6b-61871ef803a2/volumes" Oct 30 02:27:45.005374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.005329 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv"] Oct 30 02:27:45.007334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.007298 6275 cpu_manager.go:414] "containerMap: removing container" podUID="fdca2fbc-8f98-42b9-ae6b-61871ef803a2" containerName="sim" Oct 30 02:27:45.007334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.007333 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdca2fbc-8f98-42b9-ae6b-61871ef803a2" containerName="sim" Oct 30 02:27:45.007657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.007641 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdca2fbc-8f98-42b9-ae6b-61871ef803a2" containerName="sim" Oct 30 02:27:45.012761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.012735 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.034775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.034746 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d207e8\"" Oct 30 02:27:45.040653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.040618 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv"] Oct 30 02:27:45.099177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.099145 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.099177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.099182 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wbb\" (UniqueName: \"kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb\") pod \"payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.231153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.229970 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.231153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.230026 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wbb\" (UniqueName: \"kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb\") pod \"payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.235142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.231528 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.287900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.287866 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wbb\" (UniqueName: \"kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb\") pod \"payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.332970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.332932 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:45.708768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:45.708739 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv"] Oct 30 02:27:45.710992 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:27:45.710965 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7b7dbb_0fa2_43b1_a674_48a8a6f6c1d1.slice/crio-cbee08d158a7072dad7c5e645603477c76d9e259979d197b9363e71a24266bcb WatchSource:0}: Error finding container cbee08d158a7072dad7c5e645603477c76d9e259979d197b9363e71a24266bcb: Status 404 returned error can't find the container with id cbee08d158a7072dad7c5e645603477c76d9e259979d197b9363e71a24266bcb Oct 30 02:27:46.108002 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:46.107954 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" event={"ID":"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1","Type":"ContainerStarted","Data":"f1dbe7cde9315b85cd36f5f7212efb4556586444d78aa711351c33fbd2ba6811"} Oct 30 02:27:46.108443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:46.108033 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" event={"ID":"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1","Type":"ContainerStarted","Data":"cbee08d158a7072dad7c5e645603477c76d9e259979d197b9363e71a24266bcb"} Oct 30 02:27:46.108443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:46.108060 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:27:46.148671 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:46.148619 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" podStartSLOduration=2.148606814 podStartE2EDuration="2.148606814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:46.145034824 +0000 UTC m=+3494.536842458" watchObservedRunningTime="2026-10-30 02:27:46.148606814 +0000 UTC m=+3494.540414485" Oct 30 02:27:46.698218 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:27:46.698184 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:27:57.119164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:27:57.119100 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:28:16.698718 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:28:16.698637 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:28:23.920871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:23.920833 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n"] Oct 30 02:28:23.932727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:23.932699 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:23.936657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:23.934923 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n"] Oct 30 02:28:23.948769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:23.948730 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52d131\"" Oct 30 02:28:23.956626 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:23.956582 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52d131-dockercfg-wg65b\"" Oct 30 02:28:24.041346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.041279 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh\") pod \"payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.042288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.042247 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.143879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.143833 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh\") pod \"payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.144115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.144009 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.144934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.144899 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.168704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.168678 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh\") pod \"payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.270295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.270264 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:24.590111 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:24.590082 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n"] Oct 30 02:28:24.591541 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:28:24.591513 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52d08b9_a76e_4c95_a64d_3b2dbe73428d.slice/crio-905647571c475a38cc076d9008ed2acbe12435ad9e8561e11419a8c95f8ba9d2 WatchSource:0}: Error finding container 905647571c475a38cc076d9008ed2acbe12435ad9e8561e11419a8c95f8ba9d2: Status 404 returned error can't find the container with id 905647571c475a38cc076d9008ed2acbe12435ad9e8561e11419a8c95f8ba9d2 Oct 30 02:28:25.560951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:25.560917 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" event={"ID":"a52d08b9-a76e-4c95-a64d-3b2dbe73428d","Type":"ContainerStarted","Data":"29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b"} Oct 30 02:28:25.561396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:25.560975 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" event={"ID":"a52d08b9-a76e-4c95-a64d-3b2dbe73428d","Type":"ContainerStarted","Data":"905647571c475a38cc076d9008ed2acbe12435ad9e8561e11419a8c95f8ba9d2"} Oct 30 02:28:25.561396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:25.561014 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:25.608370 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:25.608309 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" podStartSLOduration=2.608293907 podStartE2EDuration="2.608293907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:28:25.603693968 +0000 UTC m=+3533.995501602" watchObservedRunningTime="2026-10-30 02:28:25.608293907 +0000 UTC m=+3534.000101539" Oct 30 02:28:36.574866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:36.574828 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:28:37.648489 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.648453 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn"] Oct 30 02:28:37.654796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.654770 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.670098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.670071 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d207e8-serving-cert\"" Oct 30 02:28:37.686210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.686175 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn"] Oct 30 02:28:37.716402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.716372 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.716569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.716551 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.716633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.716603 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vh26\" (UniqueName: \"kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.716717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.716690 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.831424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.831382 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.854765 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.854731 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vh26\" (UniqueName: \"kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.859511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.859480 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.866171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.856805 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.866436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.860644 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.867034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.866994 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:37.867380 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:28:37.867305 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-serving-cert: secret "maas-api-e2e-inf-b-d207e8-serving-cert" not found Oct 30 02:28:37.867483 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:28:37.867438 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls podName:d0bd2b42-c8ae-4f03-aaec-766c64b870be nodeName:}" failed. No retries permitted until 2026-10-30 02:28:38.367415732 +0000 UTC m=+3546.759223359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls") pod "maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" (UID: "d0bd2b42-c8ae-4f03-aaec-766c64b870be") : secret "maas-api-e2e-inf-b-d207e8-serving-cert" not found Oct 30 02:28:37.887156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:37.887111 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vh26\" (UniqueName: \"kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:38.374646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:38.374616 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:38.377206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:38.377180 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\") pod \"maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:38.571038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:38.570998 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:38.926726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:38.926692 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn"] Oct 30 02:28:38.928817 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:28:38.928783 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bd2b42_c8ae_4f03_aaec_766c64b870be.slice/crio-64c87a27ff4f3613bf548193b4e05639aae1f869b08eb5c71652014aabb61887 WatchSource:0}: Error finding container 64c87a27ff4f3613bf548193b4e05639aae1f869b08eb5c71652014aabb61887: Status 404 returned error can't find the container with id 64c87a27ff4f3613bf548193b4e05639aae1f869b08eb5c71652014aabb61887 Oct 30 02:28:39.688805 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:39.688769 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" event={"ID":"d0bd2b42-c8ae-4f03-aaec-766c64b870be","Type":"ContainerStarted","Data":"b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba"} Oct 30 02:28:39.689018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:39.688825 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" event={"ID":"d0bd2b42-c8ae-4f03-aaec-766c64b870be","Type":"ContainerStarted","Data":"64c87a27ff4f3613bf548193b4e05639aae1f869b08eb5c71652014aabb61887"} Oct 30 02:28:39.689018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:39.688898 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:39.735467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:39.735408 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" podStartSLOduration=2.4265761279999998 podStartE2EDuration="2.735387559s" totalImagesPullingTime="308.811431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:28:39.734588299 +0000 UTC m=+3548.126395932" watchObservedRunningTime="2026-10-30 02:28:39.735387559 +0000 UTC m=+3548.127195194" Oct 30 02:28:45.703047 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:28:45.703019 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:28:46.698845 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:28:46.698807 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:29:16.701639 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:29:16.701596 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:29:30.157508 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.152301 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz"] Oct 30 02:29:30.159772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.159732 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.183884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.183843 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4593af\"/\"kube-root-ca.crt\"" Oct 30 02:29:30.184067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.183898 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4593af\"/\"default-dockercfg-nd4gc\"" Oct 30 02:29:30.184067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.184014 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4593af\"/\"test-model-4593af-kserve-self-signed-certs\"" Oct 30 02:29:30.184252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.184102 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4593af\"/\"openshift-service-ca.crt\"" Oct 30 02:29:30.189726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.189694 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz"] Oct 30 02:29:30.260212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.260180 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.260393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.260222 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.260442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.260382 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.260484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.260461 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcjj9\" (UniqueName: \"kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.260629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.260612 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.260683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.260666 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.361980 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362026 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362085 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362175 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjj9\" (UniqueName: \"kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362327 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362315 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362365 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362483 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362674 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.362770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.362750 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.364427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.364400 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.364901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.364877 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.461191 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.452955 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjj9\" (UniqueName: \"kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9\") pod \"test-model-4593af-kserve-59c57964dc-sbngz\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.484146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.479419 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:30.845749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:30.845718 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz"] Oct 30 02:29:30.849081 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:29:30.849053 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdb8202b_d600_46af_993c_e7d76fa790cd.slice/crio-46e037e1a61c11ce7d7c0ae6b7454669cdc0028760487565266bb9fd7bbf9420 WatchSource:0}: Error finding container 46e037e1a61c11ce7d7c0ae6b7454669cdc0028760487565266bb9fd7bbf9420: Status 404 returned error can't find the container with id 46e037e1a61c11ce7d7c0ae6b7454669cdc0028760487565266bb9fd7bbf9420 Oct 30 02:29:31.213602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:31.213512 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" event={"ID":"cdb8202b-d600-46af-993c-e7d76fa790cd","Type":"ContainerStarted","Data":"a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5"} Oct 30 02:29:31.213602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:31.213560 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" event={"ID":"cdb8202b-d600-46af-993c-e7d76fa790cd","Type":"ContainerStarted","Data":"46e037e1a61c11ce7d7c0ae6b7454669cdc0028760487565266bb9fd7bbf9420"} Oct 30 02:29:31.259028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:31.258956 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" podStartSLOduration=1.2589354560000001 podStartE2EDuration="1.258935456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:31.251612149 +0000 UTC m=+3599.643419783" watchObservedRunningTime="2026-10-30 02:29:31.258935456 +0000 UTC m=+3599.650743091" Oct 30 02:29:31.480501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:31.480421 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:32.521648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:32.521616 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:29:32.528620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:32.528594 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:29:34.050156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.050096 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf"] Oct 30 02:29:34.056861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.056836 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.072568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.072530 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fddb7\"/\"kube-root-ca.crt\"" Oct 30 02:29:34.083704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.083673 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fddb7\"/\"default-dockercfg-r4z22\"" Oct 30 02:29:34.090277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.090235 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fddb7\"/\"test-model-1fddb7-kserve-self-signed-certs\"" Oct 30 02:29:34.095974 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.095943 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf"] Oct 30 02:29:34.181848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.181810 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fddb7\"/\"openshift-service-ca.crt\"" Oct 30 02:29:34.241543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.241509 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.241739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.241676 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.241797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.241747 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.241797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.241791 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.241886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.241851 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvrq\" (UniqueName: \"kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.241928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.241893 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343055 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343115 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343225 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvrq\" (UniqueName: \"kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343293 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343437 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343601 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343607 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343656 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.343828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.343708 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.346194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.346161 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.346322 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.346203 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.423148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.410904 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvrq\" (UniqueName: \"kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq\") pod \"test-model-1fddb7-kserve-57f44c748c-jvszf\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:34.671661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:34.671571 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:35.021712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:35.021683 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf"] Oct 30 02:29:35.023545 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:29:35.023518 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aba3b2f_41b1_4a30_ac86_8e34829317de.slice/crio-d490bbd9b4071ea67ac9f269d9c6687d97b587a62d0d462210fff63ae06f2c9c WatchSource:0}: Error finding container d490bbd9b4071ea67ac9f269d9c6687d97b587a62d0d462210fff63ae06f2c9c: Status 404 returned error can't find the container with id d490bbd9b4071ea67ac9f269d9c6687d97b587a62d0d462210fff63ae06f2c9c Oct 30 02:29:35.247246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:35.247209 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" event={"ID":"1aba3b2f-41b1-4a30-ac86-8e34829317de","Type":"ContainerStarted","Data":"5dffc952a5cab9d7168a361118c1be49f017eb79e7a9f206ba2b7d0ba2213b03"} Oct 30 02:29:35.247734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:35.247261 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" event={"ID":"1aba3b2f-41b1-4a30-ac86-8e34829317de","Type":"ContainerStarted","Data":"d490bbd9b4071ea67ac9f269d9c6687d97b587a62d0d462210fff63ae06f2c9c"} Oct 30 02:29:35.288736 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:35.288627 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" podStartSLOduration=2.288598271 podStartE2EDuration="2.288598271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:35.286022329 +0000 UTC m=+3603.677829963" watchObservedRunningTime="2026-10-30 02:29:35.288598271 +0000 UTC m=+3603.680405904" Oct 30 02:29:35.672168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:35.672061 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:40.480492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:40.480454 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:40.493220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:40.493191 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:41.309754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:41.309726 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:29:44.672079 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:44.671993 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:44.685620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:44.685590 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:45.395844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:45.395809 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:29:46.698654 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:29:46.698618 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:29:48.008289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.008247 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5"] Oct 30 02:29:48.013733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.013702 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.038978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.038941 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c75203\"/\"test-model-c75203-kserve-self-signed-certs\"" Oct 30 02:29:48.038978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.038948 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c75203\"/\"openshift-service-ca.crt\"" Oct 30 02:29:48.039277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.039046 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c75203\"/\"default-dockercfg-gnlqk\"" Oct 30 02:29:48.039277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.039235 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c75203\"/\"kube-root-ca.crt\"" Oct 30 02:29:48.044333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.044312 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5"] Oct 30 02:29:48.119812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.119768 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tng\" (UniqueName: \"kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.120048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.120001 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.120154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.120056 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.120225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.120198 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.120267 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.120249 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.120418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.120394 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222248 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222302 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222341 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222369 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222491 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222482 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222629 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tng\" (UniqueName: \"kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222677 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222752 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.222907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.222817 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.225012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.224976 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.225181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.225134 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.293153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.290973 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tng\" (UniqueName: \"kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng\") pod \"test-model-c75203-kserve-5775644647-87jq5\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.341219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.340494 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:48.728534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:48.716789 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5"] Oct 30 02:29:49.428400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:49.428360 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" event={"ID":"097a9ebe-4d14-49a7-ab68-cf45d7e66abc","Type":"ContainerStarted","Data":"a2f643b56089aed2b081db5015f85f8367a5f6fe9ebb4fb8ea24735c92f048db"} Oct 30 02:29:49.428400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:49.428406 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" event={"ID":"097a9ebe-4d14-49a7-ab68-cf45d7e66abc","Type":"ContainerStarted","Data":"8540041d1395c1295052a447544e1b6597c215620013f85fe106c122e7ec43f3"} Oct 30 02:29:49.467838 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:49.467778 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" podStartSLOduration=2.467765513 podStartE2EDuration="2.467765513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:49.466764763 +0000 UTC m=+3617.858572396" watchObservedRunningTime="2026-10-30 02:29:49.467765513 +0000 UTC m=+3617.859573166" Oct 30 02:29:50.343233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:50.343192 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:58.342692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:58.342654 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:58.355561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:58.355532 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:29:58.515336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:29:58.515307 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:30:00.254694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.254656 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb"] Oct 30 02:30:00.262414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.262384 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.289390 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.288722 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:30:00.333615 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.333580 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb"] Oct 30 02:30:00.365225 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.365191 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8"] Oct 30 02:30:00.378170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.378139 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw"] Oct 30 02:30:00.380871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.378465 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.389039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.389006 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2x2n\" (UniqueName: \"kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.389316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.389295 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.396356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.396327 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8"] Oct 30 02:30:00.396695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.396634 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.403155 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.403110 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw"] Oct 30 02:30:00.410112 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.410092 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5"] Oct 30 02:30:00.416301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.416274 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.418924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.418899 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq"] Oct 30 02:30:00.425928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.425904 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.431888 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.431866 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p"] Oct 30 02:30:00.437619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.437599 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5"] Oct 30 02:30:00.438744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.438324 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.442613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.442592 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq"] Oct 30 02:30:00.452667 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.452637 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p"] Oct 30 02:30:00.492571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492534 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27gc\" (UniqueName: \"kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.492845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492611 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5gq\" (UniqueName: \"kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.492845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492669 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2x2n\" (UniqueName: \"kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.492845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492701 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.492845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492736 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpgwp\" (UniqueName: \"kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.493086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492893 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.493086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.492943 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rngc\" (UniqueName: \"kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc\") pod \"maas-api-key-cleanup-29888790-sf2lq\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.493086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.493000 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.493265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.493159 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmkt\" (UniqueName: \"kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.493352 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.493324 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.493403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.493376 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888790-sf2lq\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.493451 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.493435 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.493969 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.493936 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.517883 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.517847 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2x2n\" (UniqueName: \"kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.595527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.595491 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.595527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.595533 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888790-sf2lq\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.595800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.595582 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.595800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.595621 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27gc\" (UniqueName: \"kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.595800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.595667 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5gq\" (UniqueName: \"kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.595987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.595912 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpgwp\" (UniqueName: \"kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.596165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596110 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.596259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596191 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rngc\" (UniqueName: \"kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc\") pod \"maas-api-key-cleanup-29888790-sf2lq\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.596330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596255 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.596398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596385 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.596461 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596402 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmkt\" (UniqueName: \"kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.596524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596472 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888790-sf2lq\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.596583 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596551 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.596702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.596682 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.597059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.597036 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.599756 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.599734 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:00.619969 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.619935 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rngc\" (UniqueName: \"kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc\") pod \"maas-api-key-cleanup-29888790-sf2lq\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:00.620168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.620105 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpgwp\" (UniqueName: \"kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.621092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.621020 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmkt\" (UniqueName: \"kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.621092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.621039 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5gq\" (UniqueName: \"kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.622762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.622731 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27gc\" (UniqueName: \"kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.722162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.722108 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:00.745279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.745027 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:00.771284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.764463 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:00.803685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.803441 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:00.803685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:00.803651 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:01.328157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:01.327090 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb"] Oct 30 02:30:01.631325 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:01.631290 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerStarted","Data":"620055a04d70d259d51111e5a00d072c70f591da624168c92caee5af420c1db3"} Oct 30 02:30:01.914306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:01.914052 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8"] Oct 30 02:30:01.916439 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:30:01.916387 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade52ed4_75b4_4ea0_b78b_3e906930941f.slice/crio-d273fb0ad594aea82087889f188b76a5c9561ccfd90e996a1b965e39e2e94533 WatchSource:0}: Error finding container d273fb0ad594aea82087889f188b76a5c9561ccfd90e996a1b965e39e2e94533: Status 404 returned error can't find the container with id d273fb0ad594aea82087889f188b76a5c9561ccfd90e996a1b965e39e2e94533 Oct 30 02:30:01.976260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:01.976193 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq"] Oct 30 02:30:01.980768 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:30:01.980731 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e78579_84f3_4ceb_b9dc_463f35dabe4d.slice/crio-ba410d40099693bb3e8cfb16ffec83e5423cf844ebaa9f65fb655775d3c52067 WatchSource:0}: Error finding container ba410d40099693bb3e8cfb16ffec83e5423cf844ebaa9f65fb655775d3c52067: Status 404 returned error can't find the container with id ba410d40099693bb3e8cfb16ffec83e5423cf844ebaa9f65fb655775d3c52067 Oct 30 02:30:02.296512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.284174 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw"] Oct 30 02:30:02.311348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.311182 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p"] Oct 30 02:30:02.354639 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:30:02.354597 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cf6d88_85ea_40f9_b3fb_fad76ef7cc1c.slice/crio-461d14ea6ea7e82f905545bf815ff503b0f8cb25e03e47c73b95413ec693a66f WatchSource:0}: Error finding container 461d14ea6ea7e82f905545bf815ff503b0f8cb25e03e47c73b95413ec693a66f: Status 404 returned error can't find the container with id 461d14ea6ea7e82f905545bf815ff503b0f8cb25e03e47c73b95413ec693a66f Oct 30 02:30:02.389202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.389071 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5"] Oct 30 02:30:02.642366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.642321 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerStarted","Data":"461d14ea6ea7e82f905545bf815ff503b0f8cb25e03e47c73b95413ec693a66f"} Oct 30 02:30:02.644413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.644351 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerStarted","Data":"55e9b5c18bd6fc3c7b6d6fa7ca54290ae344b72b9388e0089e30e26d446e0f75"} Oct 30 02:30:02.646074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.646049 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerStarted","Data":"ba410d40099693bb3e8cfb16ffec83e5423cf844ebaa9f65fb655775d3c52067"} Oct 30 02:30:02.647621 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.647594 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerStarted","Data":"1ff6b8230fe29e6d1ef36683648b2341873413d60df6a88c95e63651fdfc6f48"} Oct 30 02:30:02.649317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.649290 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerStarted","Data":"d273fb0ad594aea82087889f188b76a5c9561ccfd90e996a1b965e39e2e94533"} Oct 30 02:30:02.651274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.651247 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerStarted","Data":"00232dea4b9fbbf120614eef7c7fd70246a4327f108bb97ccbb497848dadb818"} Oct 30 02:30:02.704326 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:02.704258 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" podStartSLOduration=2.704239363 podStartE2EDuration="2.704239363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:02.697897252 +0000 UTC m=+3631.089704886" watchObservedRunningTime="2026-10-30 02:30:02.704239363 +0000 UTC m=+3631.096047000" Oct 30 02:30:03.663209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.663175 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerStarted","Data":"24df68298bccac8e5c667589e2d7fa304f1cf015cfcfcbce4920aa91c72f5e8f"} Oct 30 02:30:03.665334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.665292 6275 generic.go:356] "Generic (PLEG): container finished" podID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerID="3873c5022c245b5adf700211814e4353b3c16f85658cbb5fbfaa8937ef1f1d66" exitCode=60 Oct 30 02:30:03.665471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.665381 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerDied","Data":"3873c5022c245b5adf700211814e4353b3c16f85658cbb5fbfaa8937ef1f1d66"} Oct 30 02:30:03.665693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.665598 6275 scope.go:122] "RemoveContainer" containerID="3873c5022c245b5adf700211814e4353b3c16f85658cbb5fbfaa8937ef1f1d66" Oct 30 02:30:03.667370 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.667298 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerStarted","Data":"076adbfb7549b684b4764755dd3d2422737d1c3030b6811d12fb0902217d887a"} Oct 30 02:30:03.670718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.670687 6275 generic.go:356] "Generic (PLEG): container finished" podID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerID="b0d77abea16e1e954f0eb0f1da93e94393af5f17668b929e806b14c7b4bf5348" exitCode=60 Oct 30 02:30:03.670854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.670826 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerDied","Data":"b0d77abea16e1e954f0eb0f1da93e94393af5f17668b929e806b14c7b4bf5348"} Oct 30 02:30:03.671283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.671262 6275 scope.go:122] "RemoveContainer" containerID="b0d77abea16e1e954f0eb0f1da93e94393af5f17668b929e806b14c7b4bf5348" Oct 30 02:30:03.690837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.690799 6275 generic.go:356] "Generic (PLEG): container finished" podID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerID="00232dea4b9fbbf120614eef7c7fd70246a4327f108bb97ccbb497848dadb818" exitCode=60 Oct 30 02:30:03.691168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.691136 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerDied","Data":"00232dea4b9fbbf120614eef7c7fd70246a4327f108bb97ccbb497848dadb818"} Oct 30 02:30:03.694418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.694387 6275 scope.go:122] "RemoveContainer" containerID="00232dea4b9fbbf120614eef7c7fd70246a4327f108bb97ccbb497848dadb818" Oct 30 02:30:03.705563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.705496 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerStarted","Data":"cad9a18b6ea6f58ca0326936cb4fa5c4557dca66073c138ee915cf53c9f94dbd"} Oct 30 02:30:03.716840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.716784 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" podStartSLOduration=3.716766737 podStartE2EDuration="3.716766737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:03.712276432 +0000 UTC m=+3632.104084066" watchObservedRunningTime="2026-10-30 02:30:03.716766737 +0000 UTC m=+3632.108574385" Oct 30 02:30:03.927548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:03.927408 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" podStartSLOduration=3.927384896 podStartE2EDuration="3.927384896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:03.924992915 +0000 UTC m=+3632.316800560" watchObservedRunningTime="2026-10-30 02:30:03.927384896 +0000 UTC m=+3632.319192531" Oct 30 02:30:04.011483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.011427 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" podStartSLOduration=4.011408938 podStartE2EDuration="4.011408938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.009252991 +0000 UTC m=+3632.401060625" watchObservedRunningTime="2026-10-30 02:30:04.011408938 +0000 UTC m=+3632.403216570" Oct 30 02:30:04.716084 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.715993 6275 generic.go:356] "Generic (PLEG): container finished" podID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerID="24df68298bccac8e5c667589e2d7fa304f1cf015cfcfcbce4920aa91c72f5e8f" exitCode=60 Oct 30 02:30:04.716576 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.716073 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerDied","Data":"24df68298bccac8e5c667589e2d7fa304f1cf015cfcfcbce4920aa91c72f5e8f"} Oct 30 02:30:04.716576 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.716482 6275 scope.go:122] "RemoveContainer" containerID="24df68298bccac8e5c667589e2d7fa304f1cf015cfcfcbce4920aa91c72f5e8f" Oct 30 02:30:04.718318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.718295 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerStarted","Data":"591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b"} Oct 30 02:30:04.720283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.720260 6275 generic.go:356] "Generic (PLEG): container finished" podID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerID="076adbfb7549b684b4764755dd3d2422737d1c3030b6811d12fb0902217d887a" exitCode=60 Oct 30 02:30:04.720397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.720357 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerDied","Data":"076adbfb7549b684b4764755dd3d2422737d1c3030b6811d12fb0902217d887a"} Oct 30 02:30:04.720693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.720667 6275 scope.go:122] "RemoveContainer" containerID="076adbfb7549b684b4764755dd3d2422737d1c3030b6811d12fb0902217d887a" Oct 30 02:30:04.722560 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.722539 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerStarted","Data":"6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1"} Oct 30 02:30:04.724483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.724460 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerStarted","Data":"1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999"} Oct 30 02:30:04.726709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.726685 6275 generic.go:356] "Generic (PLEG): container finished" podID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerID="cad9a18b6ea6f58ca0326936cb4fa5c4557dca66073c138ee915cf53c9f94dbd" exitCode=60 Oct 30 02:30:04.726826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.726724 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerDied","Data":"cad9a18b6ea6f58ca0326936cb4fa5c4557dca66073c138ee915cf53c9f94dbd"} Oct 30 02:30:04.727001 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.726985 6275 scope.go:122] "RemoveContainer" containerID="cad9a18b6ea6f58ca0326936cb4fa5c4557dca66073c138ee915cf53c9f94dbd" Oct 30 02:30:04.796332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.796274 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" podStartSLOduration=4.796253058 podStartE2EDuration="4.796253058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.790699715 +0000 UTC m=+3633.182507361" watchObservedRunningTime="2026-10-30 02:30:04.796253058 +0000 UTC m=+3633.188060691" Oct 30 02:30:04.869577 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:04.869522 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" podStartSLOduration=4.869506296 podStartE2EDuration="4.869506296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.866758968 +0000 UTC m=+3633.258566602" watchObservedRunningTime="2026-10-30 02:30:04.869506296 +0000 UTC m=+3633.261313929" Oct 30 02:30:05.744402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.744358 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerStarted","Data":"e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598"} Oct 30 02:30:05.744402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.744397 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerStarted","Data":"656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793"} Oct 30 02:30:05.745012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.744524 6275 generic.go:356] "Generic (PLEG): container finished" podID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerID="591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b" exitCode=60 Oct 30 02:30:05.745012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.744597 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerDied","Data":"591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b"} Oct 30 02:30:05.745012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.744647 6275 scope.go:122] "RemoveContainer" containerID="3873c5022c245b5adf700211814e4353b3c16f85658cbb5fbfaa8937ef1f1d66" Oct 30 02:30:05.745201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.745049 6275 scope.go:122] "RemoveContainer" containerID="591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b" Oct 30 02:30:05.745374 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:05.745354 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888790-sf2lq_odh-ai-gateway-infra(88e78579-84f3-4ceb-b9dc-463f35dabe4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" Oct 30 02:30:05.746693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.746665 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerStarted","Data":"95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c"} Oct 30 02:30:05.748616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.748595 6275 generic.go:356] "Generic (PLEG): container finished" podID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerID="6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1" exitCode=60 Oct 30 02:30:05.748725 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.748671 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerDied","Data":"6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1"} Oct 30 02:30:05.748959 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.748941 6275 scope.go:122] "RemoveContainer" containerID="6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1" Oct 30 02:30:05.749232 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:05.749208 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8_odh-ai-gateway-infra(ade52ed4-75b4-4ea0-b78b-3e906930941f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" Oct 30 02:30:05.750699 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.750675 6275 generic.go:356] "Generic (PLEG): container finished" podID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerID="1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999" exitCode=60 Oct 30 02:30:05.750794 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.750703 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerDied","Data":"1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999"} Oct 30 02:30:05.750947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.750921 6275 scope.go:122] "RemoveContainer" containerID="1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999" Oct 30 02:30:05.751266 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:05.751242 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb_odh-ai-gateway-infra(e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" Oct 30 02:30:05.760909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.760892 6275 scope.go:122] "RemoveContainer" containerID="b0d77abea16e1e954f0eb0f1da93e94393af5f17668b929e806b14c7b4bf5348" Oct 30 02:30:05.775988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:05.775965 6275 scope.go:122] "RemoveContainer" containerID="00232dea4b9fbbf120614eef7c7fd70246a4327f108bb97ccbb497848dadb818" Oct 30 02:30:06.760726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.760695 6275 generic.go:356] "Generic (PLEG): container finished" podID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerID="e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598" exitCode=60 Oct 30 02:30:06.761417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.760785 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerDied","Data":"e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598"} Oct 30 02:30:06.761417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.760838 6275 scope.go:122] "RemoveContainer" containerID="cad9a18b6ea6f58ca0326936cb4fa5c4557dca66073c138ee915cf53c9f94dbd" Oct 30 02:30:06.761417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.761153 6275 scope.go:122] "RemoveContainer" containerID="e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598" Oct 30 02:30:06.761638 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:06.761581 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw_odh-ai-gateway-infra(f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" Oct 30 02:30:06.764092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.764069 6275 generic.go:356] "Generic (PLEG): container finished" podID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerID="656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793" exitCode=60 Oct 30 02:30:06.764210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.764158 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerDied","Data":"656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793"} Oct 30 02:30:06.764547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.764526 6275 scope.go:122] "RemoveContainer" containerID="656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793" Oct 30 02:30:06.764938 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:06.764914 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5_odh-ai-gateway-infra(5902787a-e31b-4413-ae35-9c2a6f3eaccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" Oct 30 02:30:06.766400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.766376 6275 scope.go:122] "RemoveContainer" containerID="591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b" Oct 30 02:30:06.766739 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:06.766719 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888790-sf2lq_odh-ai-gateway-infra(88e78579-84f3-4ceb-b9dc-463f35dabe4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" Oct 30 02:30:06.768037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.768014 6275 generic.go:356] "Generic (PLEG): container finished" podID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerID="95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c" exitCode=60 Oct 30 02:30:06.768146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.768045 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerDied","Data":"95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c"} Oct 30 02:30:06.768399 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.768384 6275 scope.go:122] "RemoveContainer" containerID="95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c" Oct 30 02:30:06.768667 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:06.768645 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p_odh-ai-gateway-infra(408d2ae6-a8a5-4931-9e2b-61d033a57a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" Oct 30 02:30:06.770192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.770176 6275 scope.go:122] "RemoveContainer" containerID="6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1" Oct 30 02:30:06.770486 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:06.770468 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8_odh-ai-gateway-infra(ade52ed4-75b4-4ea0-b78b-3e906930941f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" Oct 30 02:30:06.771818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.771801 6275 scope.go:122] "RemoveContainer" containerID="1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999" Oct 30 02:30:06.772036 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:06.772019 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb_odh-ai-gateway-infra(e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" Oct 30 02:30:06.776519 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.776502 6275 scope.go:122] "RemoveContainer" containerID="24df68298bccac8e5c667589e2d7fa304f1cf015cfcfcbce4920aa91c72f5e8f" Oct 30 02:30:06.791866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:06.791845 6275 scope.go:122] "RemoveContainer" containerID="076adbfb7549b684b4764755dd3d2422737d1c3030b6811d12fb0902217d887a" Oct 30 02:30:07.781378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:07.781343 6275 scope.go:122] "RemoveContainer" containerID="95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c" Oct 30 02:30:07.781835 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:07.781741 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p_odh-ai-gateway-infra(408d2ae6-a8a5-4931-9e2b-61d033a57a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" Oct 30 02:30:07.783217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:07.783193 6275 scope.go:122] "RemoveContainer" containerID="e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598" Oct 30 02:30:07.783538 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:07.783512 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw_odh-ai-gateway-infra(f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" Oct 30 02:30:07.784769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:07.784752 6275 scope.go:122] "RemoveContainer" containerID="656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793" Oct 30 02:30:07.785023 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:07.785004 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5_odh-ai-gateway-infra(5902787a-e31b-4413-ae35-9c2a6f3eaccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" Oct 30 02:30:16.698553 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:16.698509 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:30:16.731843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:16.731814 6275 scope.go:122] "RemoveContainer" containerID="1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999" Oct 30 02:30:17.732070 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:17.732041 6275 scope.go:122] "RemoveContainer" containerID="656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793" Oct 30 02:30:17.878874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:17.878842 6275 generic.go:356] "Generic (PLEG): container finished" podID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerID="2679ff2b4a09d437a4be1f1dbe6b0eaf5f933cb4e7d0f9d79dffa78fea723b82" exitCode=60 Oct 30 02:30:17.879052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:17.878894 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerDied","Data":"2679ff2b4a09d437a4be1f1dbe6b0eaf5f933cb4e7d0f9d79dffa78fea723b82"} Oct 30 02:30:17.879052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:17.878940 6275 scope.go:122] "RemoveContainer" containerID="1a1d3b9008601045052e621a042815790a7f214f08709ab733822c6f2e1eb999" Oct 30 02:30:17.879460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:17.879436 6275 scope.go:122] "RemoveContainer" containerID="2679ff2b4a09d437a4be1f1dbe6b0eaf5f933cb4e7d0f9d79dffa78fea723b82" Oct 30 02:30:17.879911 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:17.879876 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb_odh-ai-gateway-infra(e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" Oct 30 02:30:18.735208 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.731606 6275 scope.go:122] "RemoveContainer" containerID="e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598" Oct 30 02:30:18.735208 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.733038 6275 scope.go:122] "RemoveContainer" containerID="6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1" Oct 30 02:30:18.746065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.739770 6275 scope.go:122] "RemoveContainer" containerID="95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c" Oct 30 02:30:18.746065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.741158 6275 scope.go:122] "RemoveContainer" containerID="591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b" Oct 30 02:30:18.959405 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.959309 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb"] Oct 30 02:30:18.979420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.979376 6275 generic.go:356] "Generic (PLEG): container finished" podID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerID="f70277c90705083fc4e10b936ae6a95357fb08b74fbc02b262f777026671bc8d" exitCode=60 Oct 30 02:30:18.979677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.979655 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerDied","Data":"f70277c90705083fc4e10b936ae6a95357fb08b74fbc02b262f777026671bc8d"} Oct 30 02:30:18.979778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.979720 6275 scope.go:122] "RemoveContainer" containerID="656b9e65cf59126bd04c065a6986be15adecc7d4e3ab6f8fc1927a2ab68f1793" Oct 30 02:30:18.980312 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:18.980290 6275 scope.go:122] "RemoveContainer" containerID="f70277c90705083fc4e10b936ae6a95357fb08b74fbc02b262f777026671bc8d" Oct 30 02:30:18.980736 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:18.980700 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5_odh-ai-gateway-infra(5902787a-e31b-4413-ae35-9c2a6f3eaccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" Oct 30 02:30:19.396925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.396900 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:19.525214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.523811 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n\" (UniqueName: \"kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n\") pod \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " Oct 30 02:30:19.525433 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.525338 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca\") pod \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\" (UID: \"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d\") " Oct 30 02:30:19.530028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.527336 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca" pod "e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" (UID: "e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:19.541637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.541606 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n" pod "e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" (UID: "e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d"). InnerVolumeSpecName "kube-api-access-b2x2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:19.628762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.628725 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2x2n\" (UniqueName: \"kubernetes.io/projected/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-kube-api-access-b2x2n\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:19.628762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:19.628754 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:20.001366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.001259 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerStarted","Data":"06a2b6dca4edb388fc64b95b7cd59d0107b0d0574cfab96403a959a97f1d51a3"} Oct 30 02:30:20.006069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.006038 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerStarted","Data":"7f8e1e1693a2d00e7400e99793c855de02541c0ae5658b1161f7bec0c6492acc"} Oct 30 02:30:20.008438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.008414 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerStarted","Data":"0ba5cdb3b7254631055cefbb9d5273fe1ef1cd5a455c7dfa35444ca7d9476095"} Oct 30 02:30:20.010493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.010469 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerStarted","Data":"f97804ec69f6df00bfee0e9f1a7efae74d61e7d3737a68e5176f5d77689817d6"} Oct 30 02:30:20.011865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.011845 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" event={"ID":"e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d","Type":"ContainerDied","Data":"620055a04d70d259d51111e5a00d072c70f591da624168c92caee5af420c1db3"} Oct 30 02:30:20.011865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.011859 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb" Oct 30 02:30:20.012032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.011887 6275 scope.go:122] "RemoveContainer" containerID="2679ff2b4a09d437a4be1f1dbe6b0eaf5f933cb4e7d0f9d79dffa78fea723b82" Oct 30 02:30:20.043814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.043784 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5"] Oct 30 02:30:20.220041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.220008 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb"] Oct 30 02:30:20.227291 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.227249 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888790-879pb"] Oct 30 02:30:20.239872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.239841 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:20.348832 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.348799 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq\" (UniqueName: \"kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq\") pod \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " Oct 30 02:30:20.349034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.348946 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca\") pod \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\" (UID: \"5902787a-e31b-4413-ae35-9c2a6f3eaccf\") " Oct 30 02:30:20.350083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.350054 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca" pod "5902787a-e31b-4413-ae35-9c2a6f3eaccf" (UID: "5902787a-e31b-4413-ae35-9c2a6f3eaccf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:20.352424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.352392 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq" pod "5902787a-e31b-4413-ae35-9c2a6f3eaccf" (UID: "5902787a-e31b-4413-ae35-9c2a6f3eaccf"). InnerVolumeSpecName "kube-api-access-nr5gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:20.472732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.472691 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr5gq\" (UniqueName: \"kubernetes.io/projected/5902787a-e31b-4413-ae35-9c2a6f3eaccf-kube-api-access-nr5gq\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:20.502153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:20.477180 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5902787a-e31b-4413-ae35-9c2a6f3eaccf-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.024388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.024357 6275 generic.go:356] "Generic (PLEG): container finished" podID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerID="0ba5cdb3b7254631055cefbb9d5273fe1ef1cd5a455c7dfa35444ca7d9476095" exitCode=60 Oct 30 02:30:21.024887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.024431 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerDied","Data":"0ba5cdb3b7254631055cefbb9d5273fe1ef1cd5a455c7dfa35444ca7d9476095"} Oct 30 02:30:21.024887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.024494 6275 scope.go:122] "RemoveContainer" containerID="95351fe87f6f8f4f13a9231b1a55ea10910e140019bc0dd902f63ecee823342c" Oct 30 02:30:21.025035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.024921 6275 scope.go:122] "RemoveContainer" containerID="0ba5cdb3b7254631055cefbb9d5273fe1ef1cd5a455c7dfa35444ca7d9476095" Oct 30 02:30:21.025327 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:21.025298 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p_odh-ai-gateway-infra(408d2ae6-a8a5-4931-9e2b-61d033a57a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" Oct 30 02:30:21.027018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.026996 6275 generic.go:356] "Generic (PLEG): container finished" podID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerID="f97804ec69f6df00bfee0e9f1a7efae74d61e7d3737a68e5176f5d77689817d6" exitCode=60 Oct 30 02:30:21.027102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.027067 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerDied","Data":"f97804ec69f6df00bfee0e9f1a7efae74d61e7d3737a68e5176f5d77689817d6"} Oct 30 02:30:21.027448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.027431 6275 scope.go:122] "RemoveContainer" containerID="f97804ec69f6df00bfee0e9f1a7efae74d61e7d3737a68e5176f5d77689817d6" Oct 30 02:30:21.027721 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:21.027703 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8_odh-ai-gateway-infra(ade52ed4-75b4-4ea0-b78b-3e906930941f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" Oct 30 02:30:21.030104 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.030086 6275 generic.go:356] "Generic (PLEG): container finished" podID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerID="06a2b6dca4edb388fc64b95b7cd59d0107b0d0574cfab96403a959a97f1d51a3" exitCode=60 Oct 30 02:30:21.030217 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.030157 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerDied","Data":"06a2b6dca4edb388fc64b95b7cd59d0107b0d0574cfab96403a959a97f1d51a3"} Oct 30 02:30:21.030568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.030544 6275 scope.go:122] "RemoveContainer" containerID="06a2b6dca4edb388fc64b95b7cd59d0107b0d0574cfab96403a959a97f1d51a3" Oct 30 02:30:21.030882 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:21.030858 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw_odh-ai-gateway-infra(f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" Oct 30 02:30:21.031421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.031403 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" Oct 30 02:30:21.031505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.031432 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5" event={"ID":"5902787a-e31b-4413-ae35-9c2a6f3eaccf","Type":"ContainerDied","Data":"55e9b5c18bd6fc3c7b6d6fa7ca54290ae344b72b9388e0089e30e26d446e0f75"} Oct 30 02:30:21.033915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.033888 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerDied","Data":"7f8e1e1693a2d00e7400e99793c855de02541c0ae5658b1161f7bec0c6492acc"} Oct 30 02:30:21.034382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.034367 6275 scope.go:122] "RemoveContainer" containerID="7f8e1e1693a2d00e7400e99793c855de02541c0ae5658b1161f7bec0c6492acc" Oct 30 02:30:21.034636 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:21.034616 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888790-sf2lq_odh-ai-gateway-infra(88e78579-84f3-4ceb-b9dc-463f35dabe4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" Oct 30 02:30:21.036181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.036157 6275 generic.go:356] "Generic (PLEG): container finished" podID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerID="7f8e1e1693a2d00e7400e99793c855de02541c0ae5658b1161f7bec0c6492acc" exitCode=60 Oct 30 02:30:21.043353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.043337 6275 scope.go:122] "RemoveContainer" containerID="6262c56b6286be4ff512f297cb568da4aa28e9047dcc580ef57575192ac14ae1" Oct 30 02:30:21.059692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.059667 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw"] Oct 30 02:30:21.074298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.074276 6275 scope.go:122] "RemoveContainer" containerID="e872c72d05263fd9e267172d8eb14664a86596bb033384745cf58b34792e4598" Oct 30 02:30:21.090099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.090078 6275 scope.go:122] "RemoveContainer" containerID="f70277c90705083fc4e10b936ae6a95357fb08b74fbc02b262f777026671bc8d" Oct 30 02:30:21.099033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.099006 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8"] Oct 30 02:30:21.101251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.101229 6275 scope.go:122] "RemoveContainer" containerID="591b21e086aed5ab73cb149608b26fa45a565cb9555e3c5289e47f7cf7d6de6b" Oct 30 02:30:21.180043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.180010 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p"] Oct 30 02:30:21.194833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.194801 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq"] Oct 30 02:30:21.218706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.218672 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5"] Oct 30 02:30:21.223892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.223857 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888790-vrfv5"] Oct 30 02:30:21.746940 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.746900 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" path="/var/lib/kubelet/pods/5902787a-e31b-4413-ae35-9c2a6f3eaccf/volumes" Oct 30 02:30:21.747610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:21.747591 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" path="/var/lib/kubelet/pods/e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d/volumes" Oct 30 02:30:22.309063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.304631 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:22.404890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.404515 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:22.409457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.409429 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:22.414650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.414622 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:22.458482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.458403 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca\") pod \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " Oct 30 02:30:22.458702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.458680 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc\" (UniqueName: \"kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc\") pod \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\" (UID: \"88e78579-84f3-4ceb-b9dc-463f35dabe4d\") " Oct 30 02:30:22.459169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.458856 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca" pod "88e78579-84f3-4ceb-b9dc-463f35dabe4d" (UID: "88e78579-84f3-4ceb-b9dc-463f35dabe4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:22.459410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.459390 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e78579-84f3-4ceb-b9dc-463f35dabe4d-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.461382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.461350 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc" pod "88e78579-84f3-4ceb-b9dc-463f35dabe4d" (UID: "88e78579-84f3-4ceb-b9dc-463f35dabe4d"). InnerVolumeSpecName "kube-api-access-6rngc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:22.560873 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.560776 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca\") pod \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " Oct 30 02:30:22.560873 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.560844 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt\" (UniqueName: \"kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt\") pod \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " Oct 30 02:30:22.561153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.560935 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc\" (UniqueName: \"kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc\") pod \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\" (UID: \"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c\") " Oct 30 02:30:22.561153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.561086 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca\") pod \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\" (UID: \"408d2ae6-a8a5-4931-9e2b-61d033a57a16\") " Oct 30 02:30:22.561876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.561180 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp\" (UniqueName: \"kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp\") pod \"ade52ed4-75b4-4ea0-b78b-3e906930941f\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " Oct 30 02:30:22.561876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.561241 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca\") pod \"ade52ed4-75b4-4ea0-b78b-3e906930941f\" (UID: \"ade52ed4-75b4-4ea0-b78b-3e906930941f\") " Oct 30 02:30:22.561876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.561251 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca" pod "f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" (UID: "f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:22.561876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.561521 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca" pod "408d2ae6-a8a5-4931-9e2b-61d033a57a16" (UID: "408d2ae6-a8a5-4931-9e2b-61d033a57a16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:22.561876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.561773 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca" pod "ade52ed4-75b4-4ea0-b78b-3e906930941f" (UID: "ade52ed4-75b4-4ea0-b78b-3e906930941f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:22.562395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.562376 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade52ed4-75b4-4ea0-b78b-3e906930941f-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.562480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.562399 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.562480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.562413 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rngc\" (UniqueName: \"kubernetes.io/projected/88e78579-84f3-4ceb-b9dc-463f35dabe4d-kube-api-access-6rngc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.562480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.562428 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408d2ae6-a8a5-4931-9e2b-61d033a57a16-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.566546 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.566526 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt" pod "408d2ae6-a8a5-4931-9e2b-61d033a57a16" (UID: "408d2ae6-a8a5-4931-9e2b-61d033a57a16"). InnerVolumeSpecName "kube-api-access-pbmkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:22.566628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.566559 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc" pod "f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" (UID: "f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c"). InnerVolumeSpecName "kube-api-access-h27gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:22.567762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.567740 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp" pod "ade52ed4-75b4-4ea0-b78b-3e906930941f" (UID: "ade52ed4-75b4-4ea0-b78b-3e906930941f"). InnerVolumeSpecName "kube-api-access-jpgwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:22.663884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.663851 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h27gc\" (UniqueName: \"kubernetes.io/projected/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c-kube-api-access-h27gc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.663884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.663880 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpgwp\" (UniqueName: \"kubernetes.io/projected/ade52ed4-75b4-4ea0-b78b-3e906930941f-kube-api-access-jpgwp\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.663884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:22.663889 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbmkt\" (UniqueName: \"kubernetes.io/projected/408d2ae6-a8a5-4931-9e2b-61d033a57a16-kube-api-access-pbmkt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:23.064658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.064630 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" Oct 30 02:30:23.064840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.064629 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq" event={"ID":"88e78579-84f3-4ceb-b9dc-463f35dabe4d","Type":"ContainerDied","Data":"ba410d40099693bb3e8cfb16ffec83e5423cf844ebaa9f65fb655775d3c52067"} Oct 30 02:30:23.064840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.064786 6275 scope.go:122] "RemoveContainer" containerID="7f8e1e1693a2d00e7400e99793c855de02541c0ae5658b1161f7bec0c6492acc" Oct 30 02:30:23.066290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.066237 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" Oct 30 02:30:23.066290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.066254 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p" event={"ID":"408d2ae6-a8a5-4931-9e2b-61d033a57a16","Type":"ContainerDied","Data":"1ff6b8230fe29e6d1ef36683648b2341873413d60df6a88c95e63651fdfc6f48"} Oct 30 02:30:23.075197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.075113 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" Oct 30 02:30:23.075447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.075421 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8" event={"ID":"ade52ed4-75b4-4ea0-b78b-3e906930941f","Type":"ContainerDied","Data":"d273fb0ad594aea82087889f188b76a5c9561ccfd90e996a1b965e39e2e94533"} Oct 30 02:30:23.077550 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.077531 6275 scope.go:122] "RemoveContainer" containerID="0ba5cdb3b7254631055cefbb9d5273fe1ef1cd5a455c7dfa35444ca7d9476095" Oct 30 02:30:23.078040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.078018 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" event={"ID":"f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c","Type":"ContainerDied","Data":"461d14ea6ea7e82f905545bf815ff503b0f8cb25e03e47c73b95413ec693a66f"} Oct 30 02:30:23.078159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.078054 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw" Oct 30 02:30:23.089425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.089402 6275 scope.go:122] "RemoveContainer" containerID="f97804ec69f6df00bfee0e9f1a7efae74d61e7d3737a68e5176f5d77689817d6" Oct 30 02:30:23.098984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.098944 6275 scope.go:122] "RemoveContainer" containerID="06a2b6dca4edb388fc64b95b7cd59d0107b0d0574cfab96403a959a97f1d51a3" Oct 30 02:30:23.156244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.156210 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p"] Oct 30 02:30:23.168178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.168149 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888790-cv55p"] Oct 30 02:30:23.204717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.204685 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq"] Oct 30 02:30:23.211478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.211451 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888790-sf2lq"] Oct 30 02:30:23.250224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.250190 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw"] Oct 30 02:30:23.257175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.257142 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888790-8n8cw"] Oct 30 02:30:23.290378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.290343 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8"] Oct 30 02:30:23.297808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.297772 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888790-5vfv8"] Oct 30 02:30:23.745865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.745827 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" path="/var/lib/kubelet/pods/408d2ae6-a8a5-4931-9e2b-61d033a57a16/volumes" Oct 30 02:30:23.746436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.746420 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" path="/var/lib/kubelet/pods/88e78579-84f3-4ceb-b9dc-463f35dabe4d/volumes" Oct 30 02:30:23.746862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.746849 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" path="/var/lib/kubelet/pods/ade52ed4-75b4-4ea0-b78b-3e906930941f/volumes" Oct 30 02:30:23.747552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:23.747536 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" path="/var/lib/kubelet/pods/f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c/volumes" Oct 30 02:30:29.513655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.513620 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp"] Oct 30 02:30:29.514630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514600 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.514630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514625 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514646 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514651 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514659 6275 cpu_manager.go:414] "containerMap: removing container" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514664 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514672 6275 cpu_manager.go:414] "containerMap: removing container" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514677 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514684 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514688 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514701 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514705 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514725 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514730 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514739 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514744 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514752 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514756 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514775 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514780 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514787 6275 cpu_manager.go:414] "containerMap: removing container" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514791 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514811 6275 cpu_manager.go:414] "containerMap: removing container" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514816 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514831 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.514823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514836 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514841 6275 cpu_manager.go:414] "containerMap: removing container" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514846 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514853 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.514857 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515012 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515023 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515036 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515046 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515054 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515065 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515075 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515085 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515143 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515159 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515170 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515178 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515191 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515198 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.515648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.515205 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a4889f-d1ee-4bb0-ab41-59dbdbc1a76d" containerName="cleanup" Oct 30 02:30:29.521379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.521357 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.527647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.527620 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt"] Oct 30 02:30:29.528716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528691 6275 cpu_manager.go:414] "containerMap: removing container" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.528716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528715 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.528890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528745 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.528890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528751 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5902787a-e31b-4413-ae35-9c2a6f3eaccf" containerName="cleanup" Oct 30 02:30:29.528890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528763 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.528890 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528769 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade52ed4-75b4-4ea0-b78b-3e906930941f" containerName="cleanup" Oct 30 02:30:29.529043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528921 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e78579-84f3-4ceb-b9dc-463f35dabe4d" containerName="cleanup" Oct 30 02:30:29.529043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528938 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="408d2ae6-a8a5-4931-9e2b-61d033a57a16" containerName="cleanup" Oct 30 02:30:29.529043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.528953 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cf6d88-85ea-40f9-b3fb-fad76ef7cc1c" containerName="cleanup" Oct 30 02:30:29.532924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.532905 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:29.545197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.545168 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp"] Oct 30 02:30:29.549326 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.549303 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-07fb25\"" Oct 30 02:30:29.558490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.558458 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt"] Oct 30 02:30:29.561373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.561344 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-07fb25-dockercfg-wfxwf\"" Oct 30 02:30:29.588665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.588628 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvpt\" (UniqueName: \"kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt\") pod \"payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:29.588842 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.588690 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cql2x\" (UniqueName: \"kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x\") pod \"payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.588927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.588894 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.589002 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.588988 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:29.713271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.711752 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cql2x\" (UniqueName: \"kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x\") pod \"payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.757145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.756773 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.757145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.756909 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:29.757145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.757136 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvpt\" (UniqueName: \"kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt\") pod \"payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:29.843390 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.792139 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:29.905288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.856307 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.925467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.925394 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cql2x\" (UniqueName: \"kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x\") pod \"payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:29.937465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:29.936716 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvpt\" (UniqueName: \"kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt\") pod \"payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:30.137750 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:30.137649 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:30.149608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:30.149576 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:30.461715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:30.461524 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp"] Oct 30 02:30:30.465538 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:30:30.465500 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1155a223_6a0e_41f5_8966_0365df3ad93a.slice/crio-c9c7085c3cc25106cd20614c4dd99421df90e8008f8e014bee8555c4fa983ab6 WatchSource:0}: Error finding container c9c7085c3cc25106cd20614c4dd99421df90e8008f8e014bee8555c4fa983ab6: Status 404 returned error can't find the container with id c9c7085c3cc25106cd20614c4dd99421df90e8008f8e014bee8555c4fa983ab6 Oct 30 02:30:30.484040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:30.484013 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt"] Oct 30 02:30:30.485020 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:30:30.484993 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8515cfeb_db30_420b_95f3_7f84e37f1e3a.slice/crio-bc961197e3608afc52f799929dc79b7822250101ed0ea1107700da716b70032e WatchSource:0}: Error finding container bc961197e3608afc52f799929dc79b7822250101ed0ea1107700da716b70032e: Status 404 returned error can't find the container with id bc961197e3608afc52f799929dc79b7822250101ed0ea1107700da716b70032e Oct 30 02:30:31.151135 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.151093 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" event={"ID":"8515cfeb-db30-420b-95f3-7f84e37f1e3a","Type":"ContainerStarted","Data":"b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8"} Oct 30 02:30:31.151645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.151171 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" event={"ID":"8515cfeb-db30-420b-95f3-7f84e37f1e3a","Type":"ContainerStarted","Data":"bc961197e3608afc52f799929dc79b7822250101ed0ea1107700da716b70032e"} Oct 30 02:30:31.151645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.151213 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:31.153185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.153151 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" event={"ID":"1155a223-6a0e-41f5-8966-0365df3ad93a","Type":"ContainerStarted","Data":"6a84e021fc6e83080fc4ff92e98286c88241356640b623d1486701840622eb60"} Oct 30 02:30:31.153308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.153198 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" event={"ID":"1155a223-6a0e-41f5-8966-0365df3ad93a","Type":"ContainerStarted","Data":"c9c7085c3cc25106cd20614c4dd99421df90e8008f8e014bee8555c4fa983ab6"} Oct 30 02:30:31.153308 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.153257 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:31.190593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.190543 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" podStartSLOduration=2.190531322 podStartE2EDuration="2.190531322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:31.188514089 +0000 UTC m=+3659.580321723" watchObservedRunningTime="2026-10-30 02:30:31.190531322 +0000 UTC m=+3659.582338955" Oct 30 02:30:31.232137 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:31.232064 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" podStartSLOduration=2.232047625 podStartE2EDuration="2.232047625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:31.227623115 +0000 UTC m=+3659.619430748" watchObservedRunningTime="2026-10-30 02:30:31.232047625 +0000 UTC m=+3659.623855258" Oct 30 02:30:42.166366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:42.166329 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:30:42.173197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:42.173169 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:30:46.698264 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:46.698222 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:30:51.584862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:51.584821 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn"] Oct 30 02:30:51.585508 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:51.585245 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" containerID="cri-o://ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874" gracePeriod=30 Oct 30 02:30:56.888345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:56.888318 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001096 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001243 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001327 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001361 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001396 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001549 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001592 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001630 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001735 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001900 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.001969 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs\") pod \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\" (UID: \"ecc49b69-9b46-40d2-94aa-9c76bd29f433\") " Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.002292 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.002761 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.002786 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.002803 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.003210 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:57.004148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.003622 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:57.006168 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.005571 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:57.007328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.007297 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:30:57.008536 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.008509 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:57.008666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.008612 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r" pod "ecc49b69-9b46-40d2-94aa-9c76bd29f433" (UID: "ecc49b69-9b46-40d2-94aa-9c76bd29f433"). InnerVolumeSpecName "kube-api-access-qc47r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:57.104880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.104845 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.104880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.104877 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc49b69-9b46-40d2-94aa-9c76bd29f433-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.105082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.104889 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.105082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.104899 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.105082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.104910 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc47r\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-kube-api-access-qc47r\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.105082 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.104919 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc49b69-9b46-40d2-94aa-9c76bd29f433-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:30:57.572679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.572645 6275 generic.go:356] "Generic (PLEG): container finished" podID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerID="ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874" exitCode=0 Oct 30 02:30:57.572869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.572694 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" Oct 30 02:30:57.572869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.572736 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" event={"ID":"ecc49b69-9b46-40d2-94aa-9c76bd29f433","Type":"ContainerDied","Data":"ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874"} Oct 30 02:30:57.572869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.572793 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn" event={"ID":"ecc49b69-9b46-40d2-94aa-9c76bd29f433","Type":"ContainerDied","Data":"ad449cf2dd30942e213fbc895fc95bf29f36c3d22c4a139be8c02d3da6a2a855"} Oct 30 02:30:57.572869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.572817 6275 scope.go:122] "RemoveContainer" containerID="ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874" Oct 30 02:30:57.584884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.584866 6275 scope.go:122] "RemoveContainer" containerID="2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c" Oct 30 02:30:57.596532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.596514 6275 scope.go:122] "RemoveContainer" containerID="ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874" Oct 30 02:30:57.596813 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:57.596792 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874\": container with ID starting with ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874 not found: ID does not exist" containerID="ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874" Oct 30 02:30:57.596872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.596822 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874"} err="failed to get container status \"ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874\": rpc error: code = NotFound desc = could not find container \"ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874\": container with ID starting with ad13ec63864aa0d7b3d8cb8578bcf1bd094610b3d54471bf4dc1864fd8f9e874 not found: ID does not exist" Oct 30 02:30:57.596872 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.596838 6275 scope.go:122] "RemoveContainer" containerID="2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c" Oct 30 02:30:57.597054 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:30:57.597034 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c\": container with ID starting with 2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c not found: ID does not exist" containerID="2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c" Oct 30 02:30:57.597095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.597061 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c"} err="failed to get container status \"2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c\": rpc error: code = NotFound desc = could not find container \"2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c\": container with ID starting with 2758e3106678762cb80105c9b47d66ccc01784c35c68c120896464490533603c not found: ID does not exist" Oct 30 02:30:57.632245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.632204 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn"] Oct 30 02:30:57.645426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.645394 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-g7dwn"] Oct 30 02:30:57.745095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:57.745056 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" path="/var/lib/kubelet/pods/ecc49b69-9b46-40d2-94aa-9c76bd29f433/volumes" Oct 30 02:30:59.337141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:59.337096 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4"] Oct 30 02:30:59.337689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:30:59.337315 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" podUID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" containerName="istio-proxy" containerID="cri-o://5afd46686a9e948c881b940274c2fdc9da528f1666840ca90649c900a3ab7ee9" gracePeriod=30 Oct 30 02:31:00.126482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:00.126434 6275 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:31:00.126668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:00.126494 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" podUID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:31:02.912332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:02.912301 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc"] Oct 30 02:31:02.912797 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:02.912511 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" containerID="cri-o://268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef" gracePeriod=30 Oct 30 02:31:04.636005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.635974 6275 generic.go:356] "Generic (PLEG): container finished" podID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" containerID="5afd46686a9e948c881b940274c2fdc9da528f1666840ca90649c900a3ab7ee9" exitCode=0 Oct 30 02:31:04.636380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.636017 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" event={"ID":"9f84fdca-7562-4edd-acc3-cf9f7c6cb183","Type":"ContainerDied","Data":"5afd46686a9e948c881b940274c2fdc9da528f1666840ca90649c900a3ab7ee9"} Oct 30 02:31:04.672248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.672222 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:31:04.774962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.774916 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.774992 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775043 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775248 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:04.775396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775311 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775386 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775480 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775513 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775556 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775618 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket\") pod \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\" (UID: \"9f84fdca-7562-4edd-acc3-cf9f7c6cb183\") " Oct 30 02:31:04.775957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.775919 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:04.776250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.776224 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:04.776404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.776356 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:04.776552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.776415 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:04.777767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.777422 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.777767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.777447 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.777767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.777462 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.777767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.777478 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.777767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.777490 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.778337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.778316 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:04.778446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.778334 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:04.778446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.778339 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "kube-api-access-57k6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:04.779039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.779020 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token" pod "9f84fdca-7562-4edd-acc3-cf9f7c6cb183" (UID: "9f84fdca-7562-4edd-acc3-cf9f7c6cb183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:04.879386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.879351 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.879386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.879383 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.879386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.879393 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57k6q\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-kube-api-access-57k6q\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:04.879386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:04.879401 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f84fdca-7562-4edd-acc3-cf9f7c6cb183-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:05.646218 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:05.646187 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" Oct 30 02:31:05.646635 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:05.646180 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4" event={"ID":"9f84fdca-7562-4edd-acc3-cf9f7c6cb183","Type":"ContainerDied","Data":"4284675f2816e5f3ee5464f2202bb390ee736b3bea4fb2dfbda261e60cf06da8"} Oct 30 02:31:05.646635 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:05.646337 6275 scope.go:122] "RemoveContainer" containerID="5afd46686a9e948c881b940274c2fdc9da528f1666840ca90649c900a3ab7ee9" Oct 30 02:31:05.706585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:05.706554 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4"] Oct 30 02:31:05.717271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:05.717245 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmchj4"] Oct 30 02:31:05.743375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:05.743343 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" path="/var/lib/kubelet/pods/9f84fdca-7562-4edd-acc3-cf9f7c6cb183/volumes" Oct 30 02:31:07.993393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:07.993364 6275 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 02:31:07.993729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:07.993424 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 02:31:08.215630 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.215601 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:31:08.241074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241043 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241095 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241152 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241185 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241282 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:08.241462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241318 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241364 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241463 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241528 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241583 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:08.241807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241642 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:08.241807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241702 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22\") pod \"f96fb179-75e5-443d-ae05-c4c42e126abd\" (UID: \"f96fb179-75e5-443d-ae05-c4c42e126abd\") " Oct 30 02:31:08.241807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241741 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:08.241807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.241803 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:08.242422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.242398 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.242422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.242425 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.242622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.242438 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96fb179-75e5-443d-ae05-c4c42e126abd-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.242622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.242447 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.242622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.242454 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.244822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.244773 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:08.244908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.244835 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:08.244908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.244871 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:08.244993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.244911 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22" pod "f96fb179-75e5-443d-ae05-c4c42e126abd" (UID: "f96fb179-75e5-443d-ae05-c4c42e126abd"). InnerVolumeSpecName "kube-api-access-llc22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:08.344537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.344499 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llc22\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-kube-api-access-llc22\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.344775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.344745 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.344775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.344769 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.344775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.344779 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96fb179-75e5-443d-ae05-c4c42e126abd-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:08.709269 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.709233 6275 generic.go:356] "Generic (PLEG): container finished" podID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerID="268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef" exitCode=0 Oct 30 02:31:08.709444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.709306 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" Oct 30 02:31:08.709444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.709322 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" event={"ID":"f96fb179-75e5-443d-ae05-c4c42e126abd","Type":"ContainerDied","Data":"268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef"} Oct 30 02:31:08.709444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.709422 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc" event={"ID":"f96fb179-75e5-443d-ae05-c4c42e126abd","Type":"ContainerDied","Data":"e179754ad67a90f82cafa5a45c5cfc638d77fef6520445e9fe2604acebf7617c"} Oct 30 02:31:08.709573 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.709459 6275 scope.go:122] "RemoveContainer" containerID="268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef" Oct 30 02:31:08.722382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.722354 6275 scope.go:122] "RemoveContainer" containerID="268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef" Oct 30 02:31:08.722674 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:31:08.722653 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef\": container with ID starting with 268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef not found: ID does not exist" containerID="268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef" Oct 30 02:31:08.722751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.722686 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef"} err="failed to get container status \"268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef\": rpc error: code = NotFound desc = could not find container \"268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef\": container with ID starting with 268055d5aedec393e6c09759ed83f41340d93a3ddfd4a364898f56d76c9a1cef not found: ID does not exist" Oct 30 02:31:08.759773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.759732 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc"] Oct 30 02:31:08.770050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:08.770001 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-dqlkc"] Oct 30 02:31:09.491744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:09.491695 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g"] Oct 30 02:31:09.492314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:09.491934 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" containerID="cri-o://2fbcf8891ad110dbdcec0902b344c1b07de211cd2d87dc798a3ccb7f61590373" gracePeriod=30 Oct 30 02:31:09.751245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:09.751162 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" path="/var/lib/kubelet/pods/f96fb179-75e5-443d-ae05-c4c42e126abd/volumes" Oct 30 02:31:10.827992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:10.827946 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:31:10.828429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:10.828015 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:31:14.761141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.761055 6275 generic.go:356] "Generic (PLEG): container finished" podID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerID="2fbcf8891ad110dbdcec0902b344c1b07de211cd2d87dc798a3ccb7f61590373" exitCode=0 Oct 30 02:31:14.761554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.761144 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" event={"ID":"ac7ecb81-e269-40d7-a976-b548e71d50b9","Type":"ContainerDied","Data":"2fbcf8891ad110dbdcec0902b344c1b07de211cd2d87dc798a3ccb7f61590373"} Oct 30 02:31:14.791549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.791523 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 02:31:14.854795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.854762 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.854814 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.854862 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.854944 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855038 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855014 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855416 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855367 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:14.855493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855411 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:14.855493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855452 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:14.855607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855492 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855550 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855702 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.855769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.855733 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy\") pod \"ac7ecb81-e269-40d7-a976-b548e71d50b9\" (UID: \"ac7ecb81-e269-40d7-a976-b548e71d50b9\") " Oct 30 02:31:14.856960 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.856936 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:14.858132 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.858090 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:14.859991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.859966 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860381 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860412 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860426 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7ecb81-e269-40d7-a976-b548e71d50b9-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860438 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860452 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860464 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.860714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.860498 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "kube-api-access-xzppb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:14.861335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.861310 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:14.864511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.864458 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo" pod "ac7ecb81-e269-40d7-a976-b548e71d50b9" (UID: "ac7ecb81-e269-40d7-a976-b548e71d50b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:14.962716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.962680 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.962716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.962714 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzppb\" (UniqueName: \"kubernetes.io/projected/ac7ecb81-e269-40d7-a976-b548e71d50b9-kube-api-access-xzppb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:14.962716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:14.962724 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7ecb81-e269-40d7-a976-b548e71d50b9-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:31:15.791063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:15.791039 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" Oct 30 02:31:15.792073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:15.792049 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g" event={"ID":"ac7ecb81-e269-40d7-a976-b548e71d50b9","Type":"ContainerDied","Data":"9b3ae207a4d9d88cc30244746f0a4dcf2212ad88fd6b2a553ecb62dfbeeedc68"} Oct 30 02:31:15.792158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:15.792106 6275 scope.go:122] "RemoveContainer" containerID="2fbcf8891ad110dbdcec0902b344c1b07de211cd2d87dc798a3ccb7f61590373" Oct 30 02:31:15.852099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:15.852067 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g"] Oct 30 02:31:15.856490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:15.856468 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-h9h8g"] Oct 30 02:31:16.699146 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:31:16.699083 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:31:17.742504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:17.742471 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" path="/var/lib/kubelet/pods/ac7ecb81-e269-40d7-a976-b548e71d50b9/volumes" Oct 30 02:31:23.638453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.638413 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf"] Oct 30 02:31:23.639531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639507 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" Oct 30 02:31:23.639531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639531 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" Oct 30 02:31:23.639646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639550 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" Oct 30 02:31:23.639646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639555 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" Oct 30 02:31:23.639646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639579 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" containerName="istio-proxy" Oct 30 02:31:23.639646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639588 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" containerName="istio-proxy" Oct 30 02:31:23.639646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639601 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" Oct 30 02:31:23.639646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639616 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" Oct 30 02:31:23.639846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639668 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" Oct 30 02:31:23.639846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639673 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" Oct 30 02:31:23.639846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639841 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7ecb81-e269-40d7-a976-b548e71d50b9" containerName="istio-proxy" Oct 30 02:31:23.639942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639855 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" Oct 30 02:31:23.639942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639867 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96fb179-75e5-443d-ae05-c4c42e126abd" containerName="istio-proxy" Oct 30 02:31:23.639942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639880 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f84fdca-7562-4edd-acc3-cf9f7c6cb183" containerName="istio-proxy" Oct 30 02:31:23.639942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.639891 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc49b69-9b46-40d2-94aa-9c76bd29f433" containerName="istio-proxy" Oct 30 02:31:23.645468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.645450 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.671320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.669598 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-07fb25-serving-cert\"" Oct 30 02:31:23.680783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.680756 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf"] Oct 30 02:31:23.719054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.719006 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzgv\" (UniqueName: \"kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.719248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.719144 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.719248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.719229 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.719371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.719347 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.824707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.824626 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.825083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.825053 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.834056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.834019 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.834653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.834624 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzgv\" (UniqueName: \"kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.835272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.831017 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.877153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.873796 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.897628 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.897508 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzgv\" (UniqueName: \"kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.899314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.899282 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls\") pod \"maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:23.962186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:23.962152 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:24.284498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:24.284473 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf"] Oct 30 02:31:24.286360 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:31:24.286331 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea512869_be10_4520_a3e1_6d1827de552f.slice/crio-31ba927c2a4d736187de7127986688bb4ffe5de0b78e1b3a6325c2a903963fbf WatchSource:0}: Error finding container 31ba927c2a4d736187de7127986688bb4ffe5de0b78e1b3a6325c2a903963fbf: Status 404 returned error can't find the container with id 31ba927c2a4d736187de7127986688bb4ffe5de0b78e1b3a6325c2a903963fbf Oct 30 02:31:24.287777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:24.287761 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:31:24.911616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:24.911577 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" event={"ID":"ea512869-be10-4520-a3e1-6d1827de552f","Type":"ContainerStarted","Data":"31ba927c2a4d736187de7127986688bb4ffe5de0b78e1b3a6325c2a903963fbf"} Oct 30 02:31:25.921542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:25.921504 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" event={"ID":"ea512869-be10-4520-a3e1-6d1827de552f","Type":"ContainerStarted","Data":"03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52"} Oct 30 02:31:25.922002 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:25.921594 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:25.963449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:25.963397 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" podStartSLOduration=2.408763165 podStartE2EDuration="2.963382992s" totalImagesPullingTime="554.619827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:31:25.960321981 +0000 UTC m=+3714.352129624" watchObservedRunningTime="2026-10-30 02:31:25.963382992 +0000 UTC m=+3714.355190625" Oct 30 02:31:31.936747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:31:31.936718 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:31:46.699653 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:31:46.699618 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:32:16.699521 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:32:16.699485 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:32:40.737161 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:40.737111 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr"] Oct 30 02:32:40.737694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:40.737327 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" containerID="cri-o://72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9" gracePeriod=30 Oct 30 02:32:45.401202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:45.401089 6275 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:32:45.401640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:45.401190 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:32:46.027988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.027962 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:32:46.136364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136328 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136529 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136579 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136609 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136664 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136697 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136749 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:32:46.136951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136774 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136862 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.136951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.136903 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy\") pod \"16341446-568b-4d49-be87-c4be1c3bdc46\" (UID: \"16341446-568b-4d49-be87-c4be1c3bdc46\") " Oct 30 02:32:46.137592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.137567 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16341446-568b-4d49-be87-c4be1c3bdc46-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.137763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.137630 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:46.137913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.137804 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:46.138008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.137974 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:46.138197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.138170 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:46.139873 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.139849 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:32:46.139996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.139867 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:32:46.140381 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.140356 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "kube-api-access-k7xdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:32:46.140818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.140792 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy" pod "16341446-568b-4d49-be87-c4be1c3bdc46" (UID: "16341446-568b-4d49-be87-c4be1c3bdc46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:46.238981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.238939 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7xdm\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-kube-api-access-k7xdm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.238981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.238972 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.238981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.238982 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.238981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.238991 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.239298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.238999 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16341446-568b-4d49-be87-c4be1c3bdc46-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.239298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.239007 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.239298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.239015 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16341446-568b-4d49-be87-c4be1c3bdc46-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.239298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.239025 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16341446-568b-4d49-be87-c4be1c3bdc46-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:32:46.672537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.672504 6275 generic.go:356] "Generic (PLEG): container finished" podID="16341446-568b-4d49-be87-c4be1c3bdc46" containerID="72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9" exitCode=0 Oct 30 02:32:46.672973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.672543 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" event={"ID":"16341446-568b-4d49-be87-c4be1c3bdc46","Type":"ContainerDied","Data":"72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9"} Oct 30 02:32:46.672973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.672612 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" event={"ID":"16341446-568b-4d49-be87-c4be1c3bdc46","Type":"ContainerDied","Data":"0ff6c4ad27c411f9e7a90eccced8fd4abef354308fc738de3c13d0f10dfb0efc"} Oct 30 02:32:46.672973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.672554 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr" Oct 30 02:32:46.672973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.672642 6275 scope.go:122] "RemoveContainer" containerID="72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9" Oct 30 02:32:46.685914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.685891 6275 scope.go:122] "RemoveContainer" containerID="72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9" Oct 30 02:32:46.686209 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:32:46.686189 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9\": container with ID starting with 72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9 not found: ID does not exist" containerID="72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9" Oct 30 02:32:46.686315 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.686217 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9"} err="failed to get container status \"72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9\": rpc error: code = NotFound desc = could not find container \"72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9\": container with ID starting with 72e4fd5d6fd0c866cb21ed98094d47d709aee3ecaa536e6236ce532841a6d9b9 not found: ID does not exist" Oct 30 02:32:46.700133 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:32:46.700094 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:32:46.742337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.736546 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr"] Oct 30 02:32:46.751819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:46.751782 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-qsmmr"] Oct 30 02:32:47.741859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:32:47.741825 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" path="/var/lib/kubelet/pods/16341446-568b-4d49-be87-c4be1c3bdc46/volumes" Oct 30 02:33:16.732236 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:33:16.716650 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:33:26.868586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:26.868544 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j"] Oct 30 02:33:31.887249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:31.887202 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" podUID="fa0caf50-9763-416c-a137-f50803282abf" containerName="payload-processing" containerID="cri-o://127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9" gracePeriod=25 Oct 30 02:33:33.485846 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:33.485799 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" podUID="fa0caf50-9763-416c-a137-f50803282abf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:33:43.013317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:43.013280 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4"] Oct 30 02:33:43.013819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:43.013563 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" containerID="cri-o://5bcc2f90f28b52da11991962bb3eb34be0ad7f88ccc5a1cefbbef27d80c66f4a" gracePeriod=30 Oct 30 02:33:43.485781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:43.485700 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" podUID="fa0caf50-9763-416c-a137-f50803282abf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:33:46.698027 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:33:46.697983 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:33:48.312048 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.312019 6275 generic.go:356] "Generic (PLEG): container finished" podID="8def214d-3691-4efe-97fb-482d22899207" containerID="5bcc2f90f28b52da11991962bb3eb34be0ad7f88ccc5a1cefbbef27d80c66f4a" exitCode=0 Oct 30 02:33:48.312418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.312099 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" event={"ID":"8def214d-3691-4efe-97fb-482d22899207","Type":"ContainerDied","Data":"5bcc2f90f28b52da11991962bb3eb34be0ad7f88ccc5a1cefbbef27d80c66f4a"} Oct 30 02:33:48.312418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.312190 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" event={"ID":"8def214d-3691-4efe-97fb-482d22899207","Type":"ContainerDied","Data":"f9810b1edbf69df346a3b85d0b3804c8f8c09d11fdc4dc5dd83daceaa9dc5e9e"} Oct 30 02:33:48.312418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.312220 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9810b1edbf69df346a3b85d0b3804c8f8c09d11fdc4dc5dd83daceaa9dc5e9e" Oct 30 02:33:48.312418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.312241 6275 scope.go:122] "RemoveContainer" containerID="79c0b88f753f0038851b4f01a63ab9e72da4748ca7b6669189831639564e6ada" Oct 30 02:33:48.339645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.339620 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:33:48.454870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.454836 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.454883 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.454999 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455198 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455097 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455198 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455180 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455288 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455396 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.455518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455412 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.455518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455425 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:33:48.455518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455453 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455484 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455516 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo\") pod \"8def214d-3691-4efe-97fb-482d22899207\" (UID: \"8def214d-3691-4efe-97fb-482d22899207\") " Oct 30 02:33:48.455803 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455701 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.455871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.455851 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.456189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.456164 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.456189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.456191 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def214d-3691-4efe-97fb-482d22899207-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.456379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.456201 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.456379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.456213 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.456379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.456226 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.457749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.457726 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "kube-api-access-bgx8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:48.457891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.457871 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:48.458083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.458065 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.458202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.458079 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo" pod "8def214d-3691-4efe-97fb-482d22899207" (UID: "8def214d-3691-4efe-97fb-482d22899207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:33:48.558023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.557940 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.558023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.557972 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def214d-3691-4efe-97fb-482d22899207-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.558023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.557981 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def214d-3691-4efe-97fb-482d22899207-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.558023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:48.557990 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgx8f\" (UniqueName: \"kubernetes.io/projected/8def214d-3691-4efe-97fb-482d22899207-kube-api-access-bgx8f\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:49.321943 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:49.321913 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4" Oct 30 02:33:49.412583 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:49.412549 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4"] Oct 30 02:33:49.416927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:49.416891 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2h6s4"] Oct 30 02:33:49.785509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:49.785476 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8def214d-3691-4efe-97fb-482d22899207" path="/var/lib/kubelet/pods/8def214d-3691-4efe-97fb-482d22899207/volumes" Oct 30 02:33:49.785871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:49.785857 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d"] Oct 30 02:33:52.216676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.216650 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:33:52.325262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.325226 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd\" (UniqueName: \"kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd\") pod \"fa0caf50-9763-416c-a137-f50803282abf\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " Oct 30 02:33:52.325466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.325318 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume\") pod \"fa0caf50-9763-416c-a137-f50803282abf\" (UID: \"fa0caf50-9763-416c-a137-f50803282abf\") " Oct 30 02:33:52.325751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.325712 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume" pod "fa0caf50-9763-416c-a137-f50803282abf" (UID: "fa0caf50-9763-416c-a137-f50803282abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:33:52.325977 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.325958 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0caf50-9763-416c-a137-f50803282abf-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:52.327564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.327543 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd" pod "fa0caf50-9763-416c-a137-f50803282abf" (UID: "fa0caf50-9763-416c-a137-f50803282abf"). InnerVolumeSpecName "kube-api-access-qsmrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:52.405406 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.405367 6275 generic.go:356] "Generic (PLEG): container finished" podID="fa0caf50-9763-416c-a137-f50803282abf" containerID="127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9" exitCode=0 Oct 30 02:33:52.405406 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.405406 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" Oct 30 02:33:52.405633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.405445 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" event={"ID":"fa0caf50-9763-416c-a137-f50803282abf","Type":"ContainerDied","Data":"127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9"} Oct 30 02:33:52.405633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.405498 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j" event={"ID":"fa0caf50-9763-416c-a137-f50803282abf","Type":"ContainerDied","Data":"08afb31da745f8ecc7f34a0fd4896e07d6400cb181f9d13a4110a783d2d94499"} Oct 30 02:33:52.405633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.405522 6275 scope.go:122] "RemoveContainer" containerID="127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9" Oct 30 02:33:52.417350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.417329 6275 scope.go:122] "RemoveContainer" containerID="127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9" Oct 30 02:33:52.417674 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:33:52.417649 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9\": container with ID starting with 127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9 not found: ID does not exist" containerID="127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9" Oct 30 02:33:52.417757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.417684 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9"} err="failed to get container status \"127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9\": rpc error: code = NotFound desc = could not find container \"127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9\": container with ID starting with 127349145ff0607230f1d5c73c87b7c1b50f69695f9705af4b4a75d525c86cc9 not found: ID does not exist" Oct 30 02:33:52.427236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.427214 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsmrd\" (UniqueName: \"kubernetes.io/projected/fa0caf50-9763-416c-a137-f50803282abf-kube-api-access-qsmrd\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:33:52.453480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.453428 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j"] Oct 30 02:33:52.458964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:52.458940 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5da6-74469dc9d6-98l6j"] Oct 30 02:33:53.743085 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:53.743052 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0caf50-9763-416c-a137-f50803282abf" path="/var/lib/kubelet/pods/fa0caf50-9763-416c-a137-f50803282abf/volumes" Oct 30 02:33:54.786644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:54.786609 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" containerID="cri-o://36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f" gracePeriod=25 Oct 30 02:33:54.953011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:54.952974 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:33:55.512457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.512419 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf"] Oct 30 02:33:55.513359 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513341 6275 cpu_manager.go:414] "containerMap: removing container" podUID="fa0caf50-9763-416c-a137-f50803282abf" containerName="payload-processing" Oct 30 02:33:55.513435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513360 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0caf50-9763-416c-a137-f50803282abf" containerName="payload-processing" Oct 30 02:33:55.513435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513382 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" Oct 30 02:33:55.513435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513390 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" Oct 30 02:33:55.513435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513427 6275 cpu_manager.go:414] "containerMap: removing container" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" Oct 30 02:33:55.513435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513434 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" Oct 30 02:33:55.513712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513578 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" Oct 30 02:33:55.513712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513595 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0caf50-9763-416c-a137-f50803282abf" containerName="payload-processing" Oct 30 02:33:55.513712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513612 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" Oct 30 02:33:55.513712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.513626 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="16341446-568b-4d49-be87-c4be1c3bdc46" containerName="istio-proxy" Oct 30 02:33:55.519815 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.519793 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.543330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.543305 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52d131-openshift-default-dockercfg-8vc6p\"" Oct 30 02:33:55.549666 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.549637 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf"] Oct 30 02:33:55.592004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.591968 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592036 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592145 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592178 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592348 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj26\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592452 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592440 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592471 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592544 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.592590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.592566 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.693963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.693926 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694003 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694264 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694187 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj26\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694331 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694284 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694333 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694351 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694408 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694420 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694453 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694494 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694565 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694631 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.694913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.694887 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.695272 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.695249 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.696792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.696743 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.697009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.696986 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.718274 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.718247 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.718499 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.718470 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj26\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:55.834664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:55.834580 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:56.178146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.177434 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf"] Oct 30 02:33:56.187783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.187741 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:33:56.187937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.187817 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:33:56.187937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.187847 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:33:56.446244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.446148 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" event={"ID":"fc204e8d-6508-4956-9e7c-b9dafe5fa84e","Type":"ContainerStarted","Data":"fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a"} Oct 30 02:33:56.446244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.446221 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" event={"ID":"fc204e8d-6508-4956-9e7c-b9dafe5fa84e","Type":"ContainerStarted","Data":"d4499f9b626e3431d839160ae228c02ff4ef959be6e2e8b9508ecbce3a9f2c0b"} Oct 30 02:33:56.489643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.489581 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podStartSLOduration=1.489566806 podStartE2EDuration="1.489566806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:33:56.487901239 +0000 UTC m=+3864.879708871" watchObservedRunningTime="2026-10-30 02:33:56.489566806 +0000 UTC m=+3864.881374438" Oct 30 02:33:56.837085 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:56.837038 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:33:57.836816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:57.836781 6275 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 02:33:57.837105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:57.836844 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 02:33:58.835467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:58.835432 6275 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 02:33:58.835656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:58.835498 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 02:33:59.834875 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:59.834842 6275 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 02:33:59.835295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:33:59.834892 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 02:34:00.835505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:00.835471 6275 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 02:34:00.835909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:00.835523 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 02:34:01.835917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:01.835882 6275 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 02:34:01.836387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:01.835943 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 02:34:02.840616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:02.840585 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:34:02.841086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:02.840900 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:34:02.841613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:02.841597 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:34:04.953215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:04.953179 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:34:14.952858 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:34:14.952827 6275 logging.go:55] [core] [Channel #14781 SubChannel #14782]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.245:50052", ServerName: "10.134.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.245:50052: connect: connection refused" Oct 30 02:34:15.087113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.087084 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:34:15.236305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.236270 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume\") pod \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " Oct 30 02:34:15.236483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.236408 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g\" (UniqueName: \"kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g\") pod \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\" (UID: \"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd\") " Oct 30 02:34:15.240305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.240251 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g" pod "a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" (UID: "a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd"). InnerVolumeSpecName "kube-api-access-dnt8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:15.240465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.240359 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume" pod "a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" (UID: "a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:15.340484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.340403 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:15.340484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.340444 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnt8g\" (UniqueName: \"kubernetes.io/projected/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd-kube-api-access-dnt8g\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:15.619356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.619271 6275 generic.go:356] "Generic (PLEG): container finished" podID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerID="36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f" exitCode=0 Oct 30 02:34:15.619356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.619336 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" Oct 30 02:34:15.619600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.619349 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" event={"ID":"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd","Type":"ContainerDied","Data":"36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f"} Oct 30 02:34:15.619600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.619433 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" event={"ID":"a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd","Type":"ContainerDied","Data":"976deae7c35d282ce56524d3b4454f622c211a4f235d66a1f73ee0f2a3246136"} Oct 30 02:34:15.619600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.619457 6275 scope.go:122] "RemoveContainer" containerID="36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f" Oct 30 02:34:15.631785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.631767 6275 scope.go:122] "RemoveContainer" containerID="36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f" Oct 30 02:34:15.632092 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:34:15.632065 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f\": container with ID starting with 36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f not found: ID does not exist" containerID="36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f" Oct 30 02:34:15.632180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.632100 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f"} err="failed to get container status \"36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f\": rpc error: code = NotFound desc = could not find container \"36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f\": container with ID starting with 36b5ff69a6f8c5f67817239b96a4955906c6081b39297cc09e3b1bb19210322f not found: ID does not exist" Oct 30 02:34:15.669292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.669257 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d"] Oct 30 02:34:15.678849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.678815 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d"] Oct 30 02:34:15.741460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.741424 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" path="/var/lib/kubelet/pods/a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd/volumes" Oct 30 02:34:15.964171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:15.963630 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d97e4c-6bc9cb45bc-5rk5d" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.245:50052\" within 1s: context deadline exceeded" Oct 30 02:34:15.964171 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:34:15.963819 6275 logging.go:55] [core] [Channel #14781 SubChannel #14782]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.245:50052", ServerName: "10.134.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.245:50052: operation was canceled" Oct 30 02:34:16.698452 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:34:16.698421 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:34:28.729360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:28.729318 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz"] Oct 30 02:34:32.583570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:32.583540 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:34:32.589746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:32.589728 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:34:35.586819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:35.586614 6275 scope.go:122] "RemoveContainer" containerID="5bcc2f90f28b52da11991962bb3eb34be0ad7f88ccc5a1cefbbef27d80c66f4a" Oct 30 02:34:36.343938 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:36.343894 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x"] Oct 30 02:34:36.375823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:36.375791 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t"] Oct 30 02:34:40.474653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:40.474615 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k"] Oct 30 02:34:40.475197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:40.474849 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" podUID="46345244-b781-451e-a9d1-5af8558860d8" containerName="maas-api" containerID="cri-o://caf76922d9daba1f1bb60acdd74a4d38f663bcf9a6183feb5583a53cb4229022" gracePeriod=30 Oct 30 02:34:41.051669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.051588 6275 generic.go:356] "Generic (PLEG): container finished" podID="46345244-b781-451e-a9d1-5af8558860d8" containerID="caf76922d9daba1f1bb60acdd74a4d38f663bcf9a6183feb5583a53cb4229022" exitCode=0 Oct 30 02:34:41.051906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.051884 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" event={"ID":"46345244-b781-451e-a9d1-5af8558860d8","Type":"ContainerDied","Data":"caf76922d9daba1f1bb60acdd74a4d38f663bcf9a6183feb5583a53cb4229022"} Oct 30 02:34:41.185334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.185302 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:34:41.295616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.295581 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls\") pod \"46345244-b781-451e-a9d1-5af8558860d8\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " Oct 30 02:34:41.295802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.295640 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca\") pod \"46345244-b781-451e-a9d1-5af8558860d8\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " Oct 30 02:34:41.295802 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.295713 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl\" (UniqueName: \"kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl\") pod \"46345244-b781-451e-a9d1-5af8558860d8\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " Oct 30 02:34:41.295897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.295869 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs\") pod \"46345244-b781-451e-a9d1-5af8558860d8\" (UID: \"46345244-b781-451e-a9d1-5af8558860d8\") " Oct 30 02:34:41.296148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.296094 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca" pod "46345244-b781-451e-a9d1-5af8558860d8" (UID: "46345244-b781-451e-a9d1-5af8558860d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:41.296713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.296686 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46345244-b781-451e-a9d1-5af8558860d8-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:41.297953 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.297929 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls" pod "46345244-b781-451e-a9d1-5af8558860d8" (UID: "46345244-b781-451e-a9d1-5af8558860d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:41.298064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.298040 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl" pod "46345244-b781-451e-a9d1-5af8558860d8" (UID: "46345244-b781-451e-a9d1-5af8558860d8"). InnerVolumeSpecName "kube-api-access-rx5tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:41.298141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.298107 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs" pod "46345244-b781-451e-a9d1-5af8558860d8" (UID: "46345244-b781-451e-a9d1-5af8558860d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:41.344482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.344445 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerName="payload-processing" containerID="cri-o://0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3" gracePeriod=25 Oct 30 02:34:41.383292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.383234 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerName="payload-pre-processing" containerID="cri-o://1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7" gracePeriod=25 Oct 30 02:34:41.409533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.409507 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:41.409616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.409536 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46345244-b781-451e-a9d1-5af8558860d8-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:41.409616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:41.409546 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5tl\" (UniqueName: \"kubernetes.io/projected/46345244-b781-451e-a9d1-5af8558860d8-kube-api-access-rx5tl\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:42.061687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:42.061659 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" Oct 30 02:34:42.061687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:42.061680 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k" event={"ID":"46345244-b781-451e-a9d1-5af8558860d8","Type":"ContainerDied","Data":"e7c3e2e32ae026f344e4b74d181843ea10913fe2ec4e46b22abac33aa89e4ac2"} Oct 30 02:34:42.062194 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:42.061742 6275 scope.go:122] "RemoveContainer" containerID="caf76922d9daba1f1bb60acdd74a4d38f663bcf9a6183feb5583a53cb4229022" Oct 30 02:34:42.110212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:42.110179 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k"] Oct 30 02:34:42.120428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:42.120395 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2943ff-7bccdbf887-p8v5k"] Oct 30 02:34:43.741781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:43.741737 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46345244-b781-451e-a9d1-5af8558860d8" path="/var/lib/kubelet/pods/46345244-b781-451e-a9d1-5af8558860d8/volumes" Oct 30 02:34:43.755032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:43.755001 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" podUID="cdb8202b-d600-46af-993c-e7d76fa790cd" containerName="main" containerID="cri-o://a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5" gracePeriod=45 Oct 30 02:34:44.059226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.059200 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:34:44.080294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.080264 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9\" (UniqueName: \"kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9\") pod \"cdb8202b-d600-46af-993c-e7d76fa790cd\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " Oct 30 02:34:44.080476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.080349 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir\") pod \"cdb8202b-d600-46af-993c-e7d76fa790cd\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " Oct 30 02:34:44.080476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.080461 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs\") pod \"cdb8202b-d600-46af-993c-e7d76fa790cd\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " Oct 30 02:34:44.080606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.080491 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home\") pod \"cdb8202b-d600-46af-993c-e7d76fa790cd\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " Oct 30 02:34:44.080663 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.080613 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir" pod "cdb8202b-d600-46af-993c-e7d76fa790cd" (UID: "cdb8202b-d600-46af-993c-e7d76fa790cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:44.080848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.080812 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home" pod "cdb8202b-d600-46af-993c-e7d76fa790cd" (UID: "cdb8202b-d600-46af-993c-e7d76fa790cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:44.081133 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.081107 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:44.081216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.081144 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:44.082708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.082683 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs" pod "cdb8202b-d600-46af-993c-e7d76fa790cd" (UID: "cdb8202b-d600-46af-993c-e7d76fa790cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:44.082829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.082797 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9" pod "cdb8202b-d600-46af-993c-e7d76fa790cd" (UID: "cdb8202b-d600-46af-993c-e7d76fa790cd"). InnerVolumeSpecName "kube-api-access-pcjj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:44.085343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.085319 6275 generic.go:356] "Generic (PLEG): container finished" podID="cdb8202b-d600-46af-993c-e7d76fa790cd" containerID="a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5" exitCode=0 Oct 30 02:34:44.085475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.085368 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" Oct 30 02:34:44.085475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.085401 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" event={"ID":"cdb8202b-d600-46af-993c-e7d76fa790cd","Type":"ContainerDied","Data":"a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5"} Oct 30 02:34:44.085475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.085440 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz" event={"ID":"cdb8202b-d600-46af-993c-e7d76fa790cd","Type":"ContainerDied","Data":"46e037e1a61c11ce7d7c0ae6b7454669cdc0028760487565266bb9fd7bbf9420"} Oct 30 02:34:44.085475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.085457 6275 scope.go:122] "RemoveContainer" containerID="a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5" Oct 30 02:34:44.115947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.115912 6275 scope.go:122] "RemoveContainer" containerID="a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5" Oct 30 02:34:44.116304 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:34:44.116284 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5\": container with ID starting with a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5 not found: ID does not exist" containerID="a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5" Oct 30 02:34:44.116411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.116313 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5"} err="failed to get container status \"a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5\": rpc error: code = NotFound desc = could not find container \"a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5\": container with ID starting with a82147f458c97c642dd899d65f4f25e634c49bdae7af8896be3724d356fedfb5 not found: ID does not exist" Oct 30 02:34:44.181994 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.181955 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm\") pod \"cdb8202b-d600-46af-993c-e7d76fa790cd\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " Oct 30 02:34:44.182195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.182007 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache\") pod \"cdb8202b-d600-46af-993c-e7d76fa790cd\" (UID: \"cdb8202b-d600-46af-993c-e7d76fa790cd\") " Oct 30 02:34:44.182329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.182299 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache" pod "cdb8202b-d600-46af-993c-e7d76fa790cd" (UID: "cdb8202b-d600-46af-993c-e7d76fa790cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:44.182586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.182556 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:44.182586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.182584 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcjj9\" (UniqueName: \"kubernetes.io/projected/cdb8202b-d600-46af-993c-e7d76fa790cd-kube-api-access-pcjj9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:44.182741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.182598 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb8202b-d600-46af-993c-e7d76fa790cd-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:44.184246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.184223 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm" pod "cdb8202b-d600-46af-993c-e7d76fa790cd" (UID: "cdb8202b-d600-46af-993c-e7d76fa790cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:44.284069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.283986 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb8202b-d600-46af-993c-e7d76fa790cd-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:34:44.501290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.490282 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz"] Oct 30 02:34:44.520154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:44.515165 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4593af/test-model-4593af-kserve-59c57964dc-sbngz"] Oct 30 02:34:45.742395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:45.742358 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb8202b-d600-46af-993c-e7d76fa790cd" path="/var/lib/kubelet/pods/cdb8202b-d600-46af-993c-e7d76fa790cd/volumes" Oct 30 02:34:46.715806 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:34:46.715762 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:34:48.864283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:48.864249 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:34:48.864697 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:48.864470 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:34:51.948305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:51.948271 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp"] Oct 30 02:34:56.949429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:56.949383 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerName="payload-processing" containerID="cri-o://2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc" gracePeriod=25 Oct 30 02:34:57.768211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:57.768161 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf"] Oct 30 02:34:58.864369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:58.864333 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:34:58.864865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:34:58.864574 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:01.477785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.477744 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:01.747853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.747822 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:35:01.784283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.784253 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd\" (UniqueName: \"kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd\") pod \"4105541b-32bc-4f12-b61c-2792bde77c6b\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " Oct 30 02:35:01.784817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.784478 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume\") pod \"4105541b-32bc-4f12-b61c-2792bde77c6b\" (UID: \"4105541b-32bc-4f12-b61c-2792bde77c6b\") " Oct 30 02:35:01.785008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.784773 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume" pod "4105541b-32bc-4f12-b61c-2792bde77c6b" (UID: "4105541b-32bc-4f12-b61c-2792bde77c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:01.793257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.792802 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4105541b-32bc-4f12-b61c-2792bde77c6b-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:01.794683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.794651 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd" pod "4105541b-32bc-4f12-b61c-2792bde77c6b" (UID: "4105541b-32bc-4f12-b61c-2792bde77c6b"). InnerVolumeSpecName "kube-api-access-rpdtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:01.802565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.802539 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:35:01.893636 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.893545 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume\") pod \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " Oct 30 02:35:01.893806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.893760 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs\" (UniqueName: \"kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs\") pod \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\" (UID: \"2ca325f8-2e91-4ef7-bc44-1d412236c8cd\") " Oct 30 02:35:01.894030 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.893999 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume" pod "2ca325f8-2e91-4ef7-bc44-1d412236c8cd" (UID: "2ca325f8-2e91-4ef7-bc44-1d412236c8cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:01.894538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.894497 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:01.894538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.894531 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdtd\" (UniqueName: \"kubernetes.io/projected/4105541b-32bc-4f12-b61c-2792bde77c6b-kube-api-access-rpdtd\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:01.896710 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.896679 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs" pod "2ca325f8-2e91-4ef7-bc44-1d412236c8cd" (UID: "2ca325f8-2e91-4ef7-bc44-1d412236c8cd"). InnerVolumeSpecName "kube-api-access-blkvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:01.996337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:01.996297 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkvs\" (UniqueName: \"kubernetes.io/projected/2ca325f8-2e91-4ef7-bc44-1d412236c8cd-kube-api-access-blkvs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:02.250667 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.250591 6275 generic.go:356] "Generic (PLEG): container finished" podID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerID="1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7" exitCode=0 Oct 30 02:35:02.250667 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.250642 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" Oct 30 02:35:02.250974 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.250688 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" event={"ID":"2ca325f8-2e91-4ef7-bc44-1d412236c8cd","Type":"ContainerDied","Data":"1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7"} Oct 30 02:35:02.250974 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.250737 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t" event={"ID":"2ca325f8-2e91-4ef7-bc44-1d412236c8cd","Type":"ContainerDied","Data":"c0b56f2da38f25236273e3fce35e781d3befee927fb10ae041675a4920b396d0"} Oct 30 02:35:02.250974 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.250758 6275 scope.go:122] "RemoveContainer" containerID="1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7" Oct 30 02:35:02.252289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.252266 6275 generic.go:356] "Generic (PLEG): container finished" podID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerID="0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3" exitCode=0 Oct 30 02:35:02.252434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.252316 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" Oct 30 02:35:02.252434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.252368 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" event={"ID":"4105541b-32bc-4f12-b61c-2792bde77c6b","Type":"ContainerDied","Data":"0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3"} Oct 30 02:35:02.252434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.252420 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x" event={"ID":"4105541b-32bc-4f12-b61c-2792bde77c6b","Type":"ContainerDied","Data":"8b269be5a94b5423e33ae3f0b56a17047fbf64289e0cbd0cab04293dafacdb1f"} Oct 30 02:35:02.265367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.265341 6275 scope.go:122] "RemoveContainer" containerID="1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7" Oct 30 02:35:02.265701 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:02.265678 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7\": container with ID starting with 1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7 not found: ID does not exist" containerID="1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7" Oct 30 02:35:02.265806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.265707 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7"} err="failed to get container status \"1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7\": rpc error: code = NotFound desc = could not find container \"1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7\": container with ID starting with 1fb8aae527bc2790f87afa05e3d1a1f8e21a037187b98444b6b3552f1d6fc6b7 not found: ID does not exist" Oct 30 02:35:02.265806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.265725 6275 scope.go:122] "RemoveContainer" containerID="0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3" Oct 30 02:35:02.281396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.281361 6275 scope.go:122] "RemoveContainer" containerID="0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3" Oct 30 02:35:02.282070 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:02.282043 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3\": container with ID starting with 0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3 not found: ID does not exist" containerID="0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3" Oct 30 02:35:02.282239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.282082 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3"} err="failed to get container status \"0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3\": rpc error: code = NotFound desc = could not find container \"0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3\": container with ID starting with 0cf1f7cf9c0463a19565466fdafef54afe0992630bed08ae183186db94e5c9c3 not found: ID does not exist" Oct 30 02:35:02.304155 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.304092 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t"] Oct 30 02:35:02.314261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.314231 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2943ff-78c97d98ff-kg26t"] Oct 30 02:35:02.339190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.339148 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x"] Oct 30 02:35:02.344521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:02.344480 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2943ff-79578bc87b-nnr4x"] Oct 30 02:35:03.742483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:03.742447 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" path="/var/lib/kubelet/pods/2ca325f8-2e91-4ef7-bc44-1d412236c8cd/volumes" Oct 30 02:35:03.742935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:03.742917 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" path="/var/lib/kubelet/pods/4105541b-32bc-4f12-b61c-2792bde77c6b/volumes" Oct 30 02:35:11.477548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:11.477508 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:12.804509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:12.804463 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" podUID="1aba3b2f-41b1-4a30-ac86-8e34829317de" containerName="main" containerID="cri-o://5dffc952a5cab9d7168a361118c1be49f017eb79e7a9f206ba2b7d0ba2213b03" gracePeriod=45 Oct 30 02:35:13.333443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:13.333101 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" podUID="1aba3b2f-41b1-4a30-ac86-8e34829317de" containerName="main" probeResult="failure" output="Get \"https://10.134.1.2:8000/ready\": dial tcp 10.134.1.2:8000: connect: connection refused" Oct 30 02:35:13.351944 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:13.351888 6275 generic.go:356] "Generic (PLEG): container finished" podID="1aba3b2f-41b1-4a30-ac86-8e34829317de" containerID="5dffc952a5cab9d7168a361118c1be49f017eb79e7a9f206ba2b7d0ba2213b03" exitCode=0 Oct 30 02:35:13.352200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:13.352132 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" event={"ID":"1aba3b2f-41b1-4a30-ac86-8e34829317de","Type":"ContainerDied","Data":"5dffc952a5cab9d7168a361118c1be49f017eb79e7a9f206ba2b7d0ba2213b03"} Oct 30 02:35:14.004187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:13.999994 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:35:14.134236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134194 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir\") pod \"1aba3b2f-41b1-4a30-ac86-8e34829317de\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " Oct 30 02:35:14.134642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134608 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs\" (UniqueName: \"kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs\") pod \"1aba3b2f-41b1-4a30-ac86-8e34829317de\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " Oct 30 02:35:14.134767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134627 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir" pod "1aba3b2f-41b1-4a30-ac86-8e34829317de" (UID: "1aba3b2f-41b1-4a30-ac86-8e34829317de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:14.134767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134723 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home\") pod \"1aba3b2f-41b1-4a30-ac86-8e34829317de\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " Oct 30 02:35:14.134897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134781 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq\" (UniqueName: \"kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq\") pod \"1aba3b2f-41b1-4a30-ac86-8e34829317de\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " Oct 30 02:35:14.134897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134847 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache\") pod \"1aba3b2f-41b1-4a30-ac86-8e34829317de\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " Oct 30 02:35:14.135018 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.134923 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm\") pod \"1aba3b2f-41b1-4a30-ac86-8e34829317de\" (UID: \"1aba3b2f-41b1-4a30-ac86-8e34829317de\") " Oct 30 02:35:14.135496 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.135467 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home" pod "1aba3b2f-41b1-4a30-ac86-8e34829317de" (UID: "1aba3b2f-41b1-4a30-ac86-8e34829317de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:14.135741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.135631 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache" pod "1aba3b2f-41b1-4a30-ac86-8e34829317de" (UID: "1aba3b2f-41b1-4a30-ac86-8e34829317de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:14.136451 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.136415 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:14.136451 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.136441 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:14.136629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.136474 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:14.139163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.139110 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm" pod "1aba3b2f-41b1-4a30-ac86-8e34829317de" (UID: "1aba3b2f-41b1-4a30-ac86-8e34829317de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:14.139880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.139748 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs" pod "1aba3b2f-41b1-4a30-ac86-8e34829317de" (UID: "1aba3b2f-41b1-4a30-ac86-8e34829317de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:14.139880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.139847 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq" pod "1aba3b2f-41b1-4a30-ac86-8e34829317de" (UID: "1aba3b2f-41b1-4a30-ac86-8e34829317de"). InnerVolumeSpecName "kube-api-access-tmvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:14.238356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.238317 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvrq\" (UniqueName: \"kubernetes.io/projected/1aba3b2f-41b1-4a30-ac86-8e34829317de-kube-api-access-tmvrq\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:14.238356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.238352 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aba3b2f-41b1-4a30-ac86-8e34829317de-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:14.238634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.238370 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aba3b2f-41b1-4a30-ac86-8e34829317de-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:14.419145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.418662 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" event={"ID":"1aba3b2f-41b1-4a30-ac86-8e34829317de","Type":"ContainerDied","Data":"d490bbd9b4071ea67ac9f269d9c6687d97b587a62d0d462210fff63ae06f2c9c"} Oct 30 02:35:14.419145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.418726 6275 scope.go:122] "RemoveContainer" containerID="5dffc952a5cab9d7168a361118c1be49f017eb79e7a9f206ba2b7d0ba2213b03" Oct 30 02:35:14.419145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.418926 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf" Oct 30 02:35:14.491957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.491926 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf"] Oct 30 02:35:14.501695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:14.501647 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1fddb7/test-model-1fddb7-kserve-57f44c748c-jvszf"] Oct 30 02:35:14.918524 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:14.918485 6275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aba3b2f_41b1_4a30_ac86_8e34829317de.slice\": RecentStats: unable to find data in memory cache]" Oct 30 02:35:15.782145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:15.771372 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aba3b2f-41b1-4a30-ac86-8e34829317de" path="/var/lib/kubelet/pods/1aba3b2f-41b1-4a30-ac86-8e34829317de/volumes" Oct 30 02:35:16.699438 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:16.699405 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:35:17.261536 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.261506 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:35:17.362283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.362246 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs\" (UniqueName: \"kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs\") pod \"ee5bde7e-c056-43f6-809c-623ea8d060a8\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " Oct 30 02:35:17.362618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.362594 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume\") pod \"ee5bde7e-c056-43f6-809c-623ea8d060a8\" (UID: \"ee5bde7e-c056-43f6-809c-623ea8d060a8\") " Oct 30 02:35:17.363046 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.363016 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume" pod "ee5bde7e-c056-43f6-809c-623ea8d060a8" (UID: "ee5bde7e-c056-43f6-809c-623ea8d060a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:17.363259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.363238 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bde7e-c056-43f6-809c-623ea8d060a8-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:17.365111 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.365083 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs" pod "ee5bde7e-c056-43f6-809c-623ea8d060a8" (UID: "ee5bde7e-c056-43f6-809c-623ea8d060a8"). InnerVolumeSpecName "kube-api-access-kr9qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:17.464843 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.464806 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr9qs\" (UniqueName: \"kubernetes.io/projected/ee5bde7e-c056-43f6-809c-623ea8d060a8-kube-api-access-kr9qs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:17.520039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.520003 6275 generic.go:356] "Generic (PLEG): container finished" podID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerID="2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc" exitCode=0 Oct 30 02:35:17.520238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.520045 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" Oct 30 02:35:17.520238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.520097 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" event={"ID":"ee5bde7e-c056-43f6-809c-623ea8d060a8","Type":"ContainerDied","Data":"2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc"} Oct 30 02:35:17.520238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.520185 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp" event={"ID":"ee5bde7e-c056-43f6-809c-623ea8d060a8","Type":"ContainerDied","Data":"dc865c7f42c74454d612a587d017d3a5f087bf3fa6aab6c309fec3a4025b007e"} Oct 30 02:35:17.520238 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.520217 6275 scope.go:122] "RemoveContainer" containerID="2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc" Oct 30 02:35:17.535723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.535330 6275 scope.go:122] "RemoveContainer" containerID="2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc" Oct 30 02:35:17.535723 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:17.535697 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc\": container with ID starting with 2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc not found: ID does not exist" containerID="2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc" Oct 30 02:35:17.535855 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.535722 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc"} err="failed to get container status \"2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc\": rpc error: code = NotFound desc = could not find container \"2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc\": container with ID starting with 2df569fa4eae59fe2252e74bb7f04509e469487c5f4bae55c8ba9cfb0bb358bc not found: ID does not exist" Oct 30 02:35:17.579142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.579092 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp"] Oct 30 02:35:17.587367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.587338 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4593af-6c44f48d6b-5rkrp"] Oct 30 02:35:17.742362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:17.742326 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" path="/var/lib/kubelet/pods/ee5bde7e-c056-43f6-809c-623ea8d060a8/volumes" Oct 30 02:35:18.698980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:18.698943 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t"] Oct 30 02:35:18.699394 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:18.699223 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" containerID="cri-o://bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc" gracePeriod=30 Oct 30 02:35:19.468926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:19.468897 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5"] Oct 30 02:35:23.989990 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:23.989922 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:35:24.095823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.095793 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.095886 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.095933 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096031 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096072 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096106 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096115 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.096482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096275 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096335 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096482 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096369 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy\") pod \"f709c917-3b09-401e-a435-c5229dc8331a\" (UID: \"f709c917-3b09-401e-a435-c5229dc8331a\") " Oct 30 02:35:24.096761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.096718 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.097334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.097308 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.097520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.097339 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.097740 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.097718 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.098350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.098323 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:24.098450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.098411 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.098859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.098837 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:24.099204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.099183 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.100483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.100458 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "kube-api-access-p2ppb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:24.100773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.100750 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo" pod "f709c917-3b09-401e-a435-c5229dc8331a" (UID: "f709c917-3b09-401e-a435-c5229dc8331a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:35:24.198998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.198966 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.198998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.198993 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2ppb\" (UniqueName: \"kubernetes.io/projected/f709c917-3b09-401e-a435-c5229dc8331a-kube-api-access-p2ppb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.198998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.199002 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f709c917-3b09-401e-a435-c5229dc8331a-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.199250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.199013 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.199250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.199022 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f709c917-3b09-401e-a435-c5229dc8331a-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.199250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.199030 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.199250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.199037 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f709c917-3b09-401e-a435-c5229dc8331a-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.247704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.247623 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6"] Oct 30 02:35:24.284421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.284379 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p"] Oct 30 02:35:24.607098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.607058 6275 generic.go:356] "Generic (PLEG): container finished" podID="f709c917-3b09-401e-a435-c5229dc8331a" containerID="bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc" exitCode=0 Oct 30 02:35:24.607333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.607312 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" event={"ID":"f709c917-3b09-401e-a435-c5229dc8331a","Type":"ContainerDied","Data":"bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc"} Oct 30 02:35:24.607404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.607373 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" event={"ID":"f709c917-3b09-401e-a435-c5229dc8331a","Type":"ContainerDied","Data":"9873973690a2ecf124050742bc85dbe2bf62745d1ad93f62b99b48f0aecdf4c1"} Oct 30 02:35:24.607462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.607404 6275 scope.go:122] "RemoveContainer" containerID="bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc" Oct 30 02:35:24.607569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.607556 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t" Oct 30 02:35:24.623179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.623148 6275 scope.go:122] "RemoveContainer" containerID="bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc" Oct 30 02:35:24.623570 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:24.623497 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc\": container with ID starting with bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc not found: ID does not exist" containerID="bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc" Oct 30 02:35:24.623570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.623534 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc"} err="failed to get container status \"bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc\": rpc error: code = NotFound desc = could not find container \"bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc\": container with ID starting with bcc75a553c419c49fa1938335b7adef2cfec33a1585ffb25f2b8fd6b2c94fbbc not found: ID does not exist" Oct 30 02:35:24.668723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.668321 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t"] Oct 30 02:35:24.672932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:24.672897 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rgk4t"] Oct 30 02:35:25.742248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:25.742215 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f709c917-3b09-401e-a435-c5229dc8331a" path="/var/lib/kubelet/pods/f709c917-3b09-401e-a435-c5229dc8331a/volumes" Oct 30 02:35:29.248179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.248136 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" podUID="5ca45faa-a462-479f-9a48-9ab128512db4" containerName="payload-processing" containerID="cri-o://944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5" gracePeriod=25 Oct 30 02:35:29.285518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.285476 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" podUID="29f620af-6b5a-43d9-b385-4160459c791f" containerName="payload-pre-processing" containerID="cri-o://c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f" gracePeriod=25 Oct 30 02:35:29.533495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.533459 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:35:29.616262 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.616227 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft\" (UniqueName: \"kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft\") pod \"5ca45faa-a462-479f-9a48-9ab128512db4\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " Oct 30 02:35:29.616467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.616290 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume\") pod \"5ca45faa-a462-479f-9a48-9ab128512db4\" (UID: \"5ca45faa-a462-479f-9a48-9ab128512db4\") " Oct 30 02:35:29.616783 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.616751 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume" pod "5ca45faa-a462-479f-9a48-9ab128512db4" (UID: "5ca45faa-a462-479f-9a48-9ab128512db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:29.616961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.616940 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca45faa-a462-479f-9a48-9ab128512db4-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:29.619177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.619107 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft" pod "5ca45faa-a462-479f-9a48-9ab128512db4" (UID: "5ca45faa-a462-479f-9a48-9ab128512db4"). InnerVolumeSpecName "kube-api-access-l7mft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:29.648333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.648304 6275 generic.go:356] "Generic (PLEG): container finished" podID="5ca45faa-a462-479f-9a48-9ab128512db4" containerID="944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5" exitCode=0 Oct 30 02:35:29.648456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.648347 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" Oct 30 02:35:29.648456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.648384 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" event={"ID":"5ca45faa-a462-479f-9a48-9ab128512db4","Type":"ContainerDied","Data":"944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5"} Oct 30 02:35:29.648542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.648461 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6" event={"ID":"5ca45faa-a462-479f-9a48-9ab128512db4","Type":"ContainerDied","Data":"e4558dc3918c036bba903755dc6efbbbf7ecd75daf7d3c56e5f4c43a63277b5c"} Oct 30 02:35:29.648542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.648483 6275 scope.go:122] "RemoveContainer" containerID="944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5" Oct 30 02:35:29.660292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.660274 6275 scope.go:122] "RemoveContainer" containerID="944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5" Oct 30 02:35:29.660546 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:29.660528 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5\": container with ID starting with 944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5 not found: ID does not exist" containerID="944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5" Oct 30 02:35:29.660592 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.660553 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5"} err="failed to get container status \"944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5\": rpc error: code = NotFound desc = could not find container \"944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5\": container with ID starting with 944f22065ad9beffe4b6cbe00a8b4ab3969d0b7e61865c807eaaa28955f0c5d5 not found: ID does not exist" Oct 30 02:35:29.695820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.695791 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6"] Oct 30 02:35:29.699771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.699746 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db01c4-5bd4b7f956-7nkt6"] Oct 30 02:35:29.718466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.718440 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mft\" (UniqueName: \"kubernetes.io/projected/5ca45faa-a462-479f-9a48-9ab128512db4-kube-api-access-l7mft\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:29.740908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:29.740884 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca45faa-a462-479f-9a48-9ab128512db4" path="/var/lib/kubelet/pods/5ca45faa-a462-479f-9a48-9ab128512db4/volumes" Oct 30 02:35:33.805747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.805713 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4"] Oct 30 02:35:33.806967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.806944 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" Oct 30 02:35:33.807024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.806972 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8def214d-3691-4efe-97fb-482d22899207" containerName="istio-proxy" Oct 30 02:35:33.807024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.806992 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerName="payload-processing" Oct 30 02:35:33.807024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807000 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerName="payload-processing" Oct 30 02:35:33.807024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807012 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" Oct 30 02:35:33.807024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807020 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807045 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5ca45faa-a462-479f-9a48-9ab128512db4" containerName="payload-processing" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807052 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca45faa-a462-479f-9a48-9ab128512db4" containerName="payload-processing" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807071 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1aba3b2f-41b1-4a30-ac86-8e34829317de" containerName="main" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807077 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aba3b2f-41b1-4a30-ac86-8e34829317de" containerName="main" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807091 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerName="payload-pre-processing" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807099 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerName="payload-pre-processing" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807138 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807145 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807163 6275 cpu_manager.go:414] "containerMap: removing container" podUID="cdb8202b-d600-46af-993c-e7d76fa790cd" containerName="main" Oct 30 02:35:33.807210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807169 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb8202b-d600-46af-993c-e7d76fa790cd" containerName="main" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807241 6275 cpu_manager.go:414] "containerMap: removing container" podUID="46345244-b781-451e-a9d1-5af8558860d8" containerName="maas-api" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807248 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46345244-b781-451e-a9d1-5af8558860d8" containerName="maas-api" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807260 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerName="payload-processing" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807267 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerName="payload-processing" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807457 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="46345244-b781-451e-a9d1-5af8558860d8" containerName="maas-api" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807472 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5bde7e-c056-43f6-809c-623ea8d060a8" containerName="payload-processing" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807484 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4105541b-32bc-4f12-b61c-2792bde77c6b" containerName="payload-processing" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807494 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ab6efa-5a2a-4018-ba8f-5d64d52d9bbd" containerName="payload-pre-processing" Oct 30 02:35:33.807518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807507 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca45faa-a462-479f-9a48-9ab128512db4" containerName="payload-processing" Oct 30 02:35:33.807792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807520 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aba3b2f-41b1-4a30-ac86-8e34829317de" containerName="main" Oct 30 02:35:33.807792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807529 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca325f8-2e91-4ef7-bc44-1d412236c8cd" containerName="payload-pre-processing" Oct 30 02:35:33.807792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807543 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f709c917-3b09-401e-a435-c5229dc8331a" containerName="istio-proxy" Oct 30 02:35:33.807792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.807552 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb8202b-d600-46af-993c-e7d76fa790cd" containerName="main" Oct 30 02:35:33.820829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.820792 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.832026 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.831981 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4"] Oct 30 02:35:33.878651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.878613 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.878853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.878660 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.878853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.878754 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.878955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.878907 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.879021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.879004 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.879075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.879052 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.879323 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.879295 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.879457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.879438 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.879530 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.879512 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.980745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980705 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.980932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980784 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.980932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980810 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.980932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980843 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.980932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980864 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.980932 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980904 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.980989 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981049 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981091 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981107 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981213 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981391 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981462 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.981903 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.981881 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.983182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.983160 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:33.983436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:33.983417 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:34.002512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.002490 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:34.002734 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.002713 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt\") pod \"maas-default-gateway-openshift-default-75d7c785f-9stc4\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:34.152205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.152116 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:34.480740 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.480657 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4"] Oct 30 02:35:34.487219 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:35:34.487186 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c955d1c_d428_4ac7_b931_4cfe83e3ca92.slice/crio-71162875e86c45fdbdfd684da08c575a552c7ae1e79037c58182a3562689621d WatchSource:0}: Error finding container 71162875e86c45fdbdfd684da08c575a552c7ae1e79037c58182a3562689621d: Status 404 returned error can't find the container with id 71162875e86c45fdbdfd684da08c575a552c7ae1e79037c58182a3562689621d Oct 30 02:35:34.490211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.490168 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:35:34.490329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.490278 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:35:34.490402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.490325 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:35:34.490650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.490617 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" podUID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" containerName="main" containerID="cri-o://a2f643b56089aed2b081db5015f85f8367a5f6fe9ebb4fb8ea24735c92f048db" gracePeriod=45 Oct 30 02:35:34.503988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.503962 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" podUID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" containerName="main" probeResult="failure" output="Get \"https://10.134.1.3:8000/ready\": dial tcp 10.134.1.3:8000: connect: connection refused" Oct 30 02:35:34.694691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.694644 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" event={"ID":"0c955d1c-d428-4ac7-b931-4cfe83e3ca92","Type":"ContainerStarted","Data":"b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a"} Oct 30 02:35:34.694913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.694714 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" event={"ID":"0c955d1c-d428-4ac7-b931-4cfe83e3ca92","Type":"ContainerStarted","Data":"71162875e86c45fdbdfd684da08c575a552c7ae1e79037c58182a3562689621d"} Oct 30 02:35:34.697812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.697784 6275 generic.go:356] "Generic (PLEG): container finished" podID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" containerID="a2f643b56089aed2b081db5015f85f8367a5f6fe9ebb4fb8ea24735c92f048db" exitCode=0 Oct 30 02:35:34.697942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.697894 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" event={"ID":"097a9ebe-4d14-49a7-ab68-cf45d7e66abc","Type":"ContainerDied","Data":"a2f643b56089aed2b081db5015f85f8367a5f6fe9ebb4fb8ea24735c92f048db"} Oct 30 02:35:34.749313 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.749186 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podStartSLOduration=1.74916451 podStartE2EDuration="1.74916451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:35:34.738226174 +0000 UTC m=+3963.130033808" watchObservedRunningTime="2026-10-30 02:35:34.74916451 +0000 UTC m=+3963.140972143" Oct 30 02:35:34.791077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.791049 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:35:34.896573 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896538 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache\") pod \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896590 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng\" (UniqueName: \"kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng\") pod \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896649 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir\") pod \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896794 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs\" (UniqueName: \"kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs\") pod \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896816 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache" pod "097a9ebe-4d14-49a7-ab68-cf45d7e66abc" (UID: "097a9ebe-4d14-49a7-ab68-cf45d7e66abc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896928 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm\") pod \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.896927 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir" pod "097a9ebe-4d14-49a7-ab68-cf45d7e66abc" (UID: "097a9ebe-4d14-49a7-ab68-cf45d7e66abc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:34.897057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.897007 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home\") pod \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\" (UID: \"097a9ebe-4d14-49a7-ab68-cf45d7e66abc\") " Oct 30 02:35:34.897511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.897258 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home" pod "097a9ebe-4d14-49a7-ab68-cf45d7e66abc" (UID: "097a9ebe-4d14-49a7-ab68-cf45d7e66abc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:34.897766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.897746 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:34.897812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.897775 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:34.897812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.897790 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:34.899108 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.899087 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng" pod "097a9ebe-4d14-49a7-ab68-cf45d7e66abc" (UID: "097a9ebe-4d14-49a7-ab68-cf45d7e66abc"). InnerVolumeSpecName "kube-api-access-x6tng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:34.899200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.899092 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs" pod "097a9ebe-4d14-49a7-ab68-cf45d7e66abc" (UID: "097a9ebe-4d14-49a7-ab68-cf45d7e66abc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:34.899302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.899285 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm" pod "097a9ebe-4d14-49a7-ab68-cf45d7e66abc" (UID: "097a9ebe-4d14-49a7-ab68-cf45d7e66abc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:34.999058 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.999030 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:34.999224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.999062 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tng\" (UniqueName: \"kubernetes.io/projected/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-kube-api-access-x6tng\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:34.999224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:34.999078 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097a9ebe-4d14-49a7-ab68-cf45d7e66abc-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:35.153060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.153024 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:35.155165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.155134 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:35.155326 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.155190 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:35.457240 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.453812 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" podUID="29f620af-6b5a-43d9-b385-4160459c791f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:35.706597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.706574 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" Oct 30 02:35:35.706597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.706585 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5" event={"ID":"097a9ebe-4d14-49a7-ab68-cf45d7e66abc","Type":"ContainerDied","Data":"8540041d1395c1295052a447544e1b6597c215620013f85fe106c122e7ec43f3"} Oct 30 02:35:35.706753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.706634 6275 scope.go:122] "RemoveContainer" containerID="a2f643b56089aed2b081db5015f85f8367a5f6fe9ebb4fb8ea24735c92f048db" Oct 30 02:35:35.770157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.770023 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5"] Oct 30 02:35:35.775961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:35.775932 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c75203/test-model-c75203-kserve-5775644647-87jq5"] Oct 30 02:35:36.158146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:36.155545 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:36.158146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:36.155607 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:37.158144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:37.154513 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:37.158144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:37.154576 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:37.766066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:37.764103 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" path="/var/lib/kubelet/pods/097a9ebe-4d14-49a7-ab68-cf45d7e66abc/volumes" Oct 30 02:35:38.163578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:38.163470 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:38.163578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:38.163551 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:39.158154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:39.154478 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:39.158154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:39.154552 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:40.156250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:40.156214 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:40.156882 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:40.156851 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:41.152562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:41.152523 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:41.152751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:41.152588 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:42.152993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:42.152909 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9stc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 02:35:42.152993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:42.152961 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 02:35:43.158587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:43.158555 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:43.159042 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:43.159026 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:43.159741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:43.159718 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:35:45.452286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:45.452245 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" podUID="29f620af-6b5a-43d9-b385-4160459c791f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:46.697874 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:46.697840 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:35:49.587226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.587197 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:35:49.662806 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.662771 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm\" (UniqueName: \"kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm\") pod \"29f620af-6b5a-43d9-b385-4160459c791f\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " Oct 30 02:35:49.662992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.662897 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume\") pod \"29f620af-6b5a-43d9-b385-4160459c791f\" (UID: \"29f620af-6b5a-43d9-b385-4160459c791f\") " Oct 30 02:35:49.663531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.663497 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume" pod "29f620af-6b5a-43d9-b385-4160459c791f" (UID: "29f620af-6b5a-43d9-b385-4160459c791f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:49.663980 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.663958 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29f620af-6b5a-43d9-b385-4160459c791f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:49.665288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.665262 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm" pod "29f620af-6b5a-43d9-b385-4160459c791f" (UID: "29f620af-6b5a-43d9-b385-4160459c791f"). InnerVolumeSpecName "kube-api-access-s6tcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:49.765353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.765317 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6tcm\" (UniqueName: \"kubernetes.io/projected/29f620af-6b5a-43d9-b385-4160459c791f-kube-api-access-s6tcm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:35:49.846385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.846303 6275 generic.go:356] "Generic (PLEG): container finished" podID="29f620af-6b5a-43d9-b385-4160459c791f" containerID="c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f" exitCode=0 Oct 30 02:35:49.846385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.846350 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" Oct 30 02:35:49.846385 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.846360 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" event={"ID":"29f620af-6b5a-43d9-b385-4160459c791f","Type":"ContainerDied","Data":"c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f"} Oct 30 02:35:49.846619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.846417 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p" event={"ID":"29f620af-6b5a-43d9-b385-4160459c791f","Type":"ContainerDied","Data":"f287e748093bfc6f1aef8a1d1f395766dc97a923d85d227f2bca2883eb112689"} Oct 30 02:35:49.846619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.846442 6275 scope.go:122] "RemoveContainer" containerID="c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f" Oct 30 02:35:49.858632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.858615 6275 scope.go:122] "RemoveContainer" containerID="c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f" Oct 30 02:35:49.858924 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:35:49.858900 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f\": container with ID starting with c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f not found: ID does not exist" containerID="c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f" Oct 30 02:35:49.858970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.858932 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f"} err="failed to get container status \"c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f\": rpc error: code = NotFound desc = could not find container \"c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f\": container with ID starting with c7361b0dd61abe198dfc02bf3f3f8897dea7a521260eacc6cc66a401c3c5706f not found: ID does not exist" Oct 30 02:35:49.896829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.896799 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p"] Oct 30 02:35:49.900624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:49.900598 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db01c4-f9bd75d8c-k2c4p"] Oct 30 02:35:51.741694 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:35:51.741664 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f620af-6b5a-43d9-b385-4160459c791f" path="/var/lib/kubelet/pods/29f620af-6b5a-43d9-b385-4160459c791f/volumes" Oct 30 02:36:02.087379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:02.087343 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l"] Oct 30 02:36:02.087922 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:02.087613 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" containerID="cri-o://35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981" gracePeriod=30 Oct 30 02:36:07.376067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.376037 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:36:07.399467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399433 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.399658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399476 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.399658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399585 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.399787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399723 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.399787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399755 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.399897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399798 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.399897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.399843 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.400069 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.400045 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.400235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.400101 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket\") pod \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\" (UID: \"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7\") " Oct 30 02:36:07.400369 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.400343 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.400776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.400749 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.400937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.400890 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.401066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.401044 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.401268 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.401248 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.401406 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.401312 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:07.402490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.402434 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.402823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.402799 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:36:07.403823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.403799 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:07.406226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.406199 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz" pod "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" (UID: "ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7"). InnerVolumeSpecName "kube-api-access-h47cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:07.512617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512581 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512619 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512634 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512650 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47cz\" (UniqueName: \"kubernetes.io/projected/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-kube-api-access-h47cz\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512665 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512679 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512693 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.512829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:07.512706 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:08.004618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.004526 6275 generic.go:356] "Generic (PLEG): container finished" podID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerID="35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981" exitCode=0 Oct 30 02:36:08.004618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.004589 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" Oct 30 02:36:08.004618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.004606 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" event={"ID":"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7","Type":"ContainerDied","Data":"35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981"} Oct 30 02:36:08.004897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.004682 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l" event={"ID":"ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7","Type":"ContainerDied","Data":"14ab3699a6ecc9b279707835da9d91ef9b98b158cd18954b60ef1f58bba9831a"} Oct 30 02:36:08.004897 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.004704 6275 scope.go:122] "RemoveContainer" containerID="35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981" Oct 30 02:36:08.016754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.016729 6275 scope.go:122] "RemoveContainer" containerID="7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b" Oct 30 02:36:08.027053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.027028 6275 scope.go:122] "RemoveContainer" containerID="35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981" Oct 30 02:36:08.027448 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:36:08.027417 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981\": container with ID starting with 35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981 not found: ID does not exist" containerID="35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981" Oct 30 02:36:08.027574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.027460 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981"} err="failed to get container status \"35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981\": rpc error: code = NotFound desc = could not find container \"35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981\": container with ID starting with 35f49008834950ce5129af277b6912f3904b40107851b6b0115dfcc3e9dcb981 not found: ID does not exist" Oct 30 02:36:08.027574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.027487 6275 scope.go:122] "RemoveContainer" containerID="7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b" Oct 30 02:36:08.027780 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:36:08.027757 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b\": container with ID starting with 7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b not found: ID does not exist" containerID="7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b" Oct 30 02:36:08.027837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.027821 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b"} err="failed to get container status \"7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b\": rpc error: code = NotFound desc = could not find container \"7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b\": container with ID starting with 7cb262621a627cca693d85cb4094e6246c7044fdb8ebadef9f3ed6d00482803b not found: ID does not exist" Oct 30 02:36:08.071584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.071552 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l"] Oct 30 02:36:08.075332 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:08.075300 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-hfz6l"] Oct 30 02:36:09.743304 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:09.743270 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" path="/var/lib/kubelet/pods/ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7/volumes" Oct 30 02:36:10.184782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:10.184745 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw"] Oct 30 02:36:13.680985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:13.679584 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns"] Oct 30 02:36:13.680985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:13.679862 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" podUID="61926654-587a-4e1c-b209-d0ef76657bca" containerName="maas-api" containerID="cri-o://8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756" gracePeriod=30 Oct 30 02:36:14.054966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.054931 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:36:14.057077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.057042 6275 generic.go:356] "Generic (PLEG): container finished" podID="61926654-587a-4e1c-b209-d0ef76657bca" containerID="8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756" exitCode=0 Oct 30 02:36:14.057294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.057221 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" event={"ID":"61926654-587a-4e1c-b209-d0ef76657bca","Type":"ContainerDied","Data":"8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756"} Oct 30 02:36:14.057294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.057275 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" event={"ID":"61926654-587a-4e1c-b209-d0ef76657bca","Type":"ContainerDied","Data":"d98ae14e2f9ee5a34edc302bf2ed580f2b3c33af5b413bdc8df5bcd387bd86b2"} Oct 30 02:36:14.057458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.057305 6275 scope.go:122] "RemoveContainer" containerID="8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756" Oct 30 02:36:14.073337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.073189 6275 scope.go:122] "RemoveContainer" containerID="8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756" Oct 30 02:36:14.084343 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:36:14.084284 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756\": container with ID starting with 8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756 not found: ID does not exist" containerID="8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756" Oct 30 02:36:14.084863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.084366 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756"} err="failed to get container status \"8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756\": rpc error: code = NotFound desc = could not find container \"8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756\": container with ID starting with 8351bcab9431920fbd785fdb1abab2fd2c2c049d1e8c68ad882088991e500756 not found: ID does not exist" Oct 30 02:36:14.148868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.148504 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs\") pod \"61926654-587a-4e1c-b209-d0ef76657bca\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " Oct 30 02:36:14.148868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.148599 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca\") pod \"61926654-587a-4e1c-b209-d0ef76657bca\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " Oct 30 02:36:14.148868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.148808 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb\" (UniqueName: \"kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb\") pod \"61926654-587a-4e1c-b209-d0ef76657bca\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " Oct 30 02:36:14.149220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.149037 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls\") pod \"61926654-587a-4e1c-b209-d0ef76657bca\" (UID: \"61926654-587a-4e1c-b209-d0ef76657bca\") " Oct 30 02:36:14.156177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.153779 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls" pod "61926654-587a-4e1c-b209-d0ef76657bca" (UID: "61926654-587a-4e1c-b209-d0ef76657bca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:36:14.156177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.154202 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca" pod "61926654-587a-4e1c-b209-d0ef76657bca" (UID: "61926654-587a-4e1c-b209-d0ef76657bca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:14.171146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.168198 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb" pod "61926654-587a-4e1c-b209-d0ef76657bca" (UID: "61926654-587a-4e1c-b209-d0ef76657bca"). InnerVolumeSpecName "kube-api-access-rdtvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:14.251344 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.251273 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs" pod "61926654-587a-4e1c-b209-d0ef76657bca" (UID: "61926654-587a-4e1c-b209-d0ef76657bca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:36:14.253552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.253519 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:14.253552 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.253554 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61926654-587a-4e1c-b209-d0ef76657bca-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:14.253757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.253568 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61926654-587a-4e1c-b209-d0ef76657bca-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:14.253757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:14.253591 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdtvb\" (UniqueName: \"kubernetes.io/projected/61926654-587a-4e1c-b209-d0ef76657bca-kube-api-access-rdtvb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:15.066548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:15.066514 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns" Oct 30 02:36:15.115103 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:15.115074 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns"] Oct 30 02:36:15.118835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:15.118809 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db01c4-6484556949-kf4ns"] Oct 30 02:36:15.185655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:15.185620 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" containerName="payload-processing" containerID="cri-o://2468873a0117ed9ef17ce3d05b508736602bb3eafc4d7231cbfd507bd3bffcbf" gracePeriod=25 Oct 30 02:36:15.740665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:15.740631 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61926654-587a-4e1c-b209-d0ef76657bca" path="/var/lib/kubelet/pods/61926654-587a-4e1c-b209-d0ef76657bca/volumes" Oct 30 02:36:16.697207 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:36:16.697169 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:36:17.681771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:17.681733 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:36:27.681209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:27.681163 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:36:28.357489 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:28.357454 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv"] Oct 30 02:36:33.358507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:33.358463 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerName="payload-pre-processing" containerID="cri-o://f1dbe7cde9315b85cd36f5f7212efb4556586444d78aa711351c33fbd2ba6811" gracePeriod=25 Oct 30 02:36:35.485422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.485394 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:36:35.613233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.613197 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx\" (UniqueName: \"kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx\") pod \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " Oct 30 02:36:35.613449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.613262 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume\") pod \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\" (UID: \"bb24450c-9ff5-45b0-98d8-721ae6b05aae\") " Oct 30 02:36:35.613622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.613597 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume" pod "bb24450c-9ff5-45b0-98d8-721ae6b05aae" (UID: "bb24450c-9ff5-45b0-98d8-721ae6b05aae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:35.613976 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.613952 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb24450c-9ff5-45b0-98d8-721ae6b05aae-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:35.615415 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.615393 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx" pod "bb24450c-9ff5-45b0-98d8-721ae6b05aae" (UID: "bb24450c-9ff5-45b0-98d8-721ae6b05aae"). InnerVolumeSpecName "kube-api-access-6wgpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:35.715548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.715475 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgpx\" (UniqueName: \"kubernetes.io/projected/bb24450c-9ff5-45b0-98d8-721ae6b05aae-kube-api-access-6wgpx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:35.716548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:35.716529 6275 scope.go:122] "RemoveContainer" containerID="2468873a0117ed9ef17ce3d05b508736602bb3eafc4d7231cbfd507bd3bffcbf" Oct 30 02:36:36.308952 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:36.308912 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" event={"ID":"bb24450c-9ff5-45b0-98d8-721ae6b05aae","Type":"ContainerDied","Data":"2468873a0117ed9ef17ce3d05b508736602bb3eafc4d7231cbfd507bd3bffcbf"} Oct 30 02:36:36.309147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:36.308963 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" Oct 30 02:36:36.309147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:36.308996 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw" event={"ID":"bb24450c-9ff5-45b0-98d8-721ae6b05aae","Type":"ContainerDied","Data":"5d2490fec434e1f6451affc9945824a7adbaa767ae07d441607472ec5dbb09fe"} Oct 30 02:36:36.386410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:36.386378 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw"] Oct 30 02:36:36.390057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:36.390028 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c75203-7bcc988958-8z9nw"] Oct 30 02:36:37.117211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:37.117169 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:36:37.765878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:37.763999 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" path="/var/lib/kubelet/pods/bb24450c-9ff5-45b0-98d8-721ae6b05aae/volumes" Oct 30 02:36:46.697948 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:36:46.697911 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:36:47.116703 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:47.116668 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:36:53.486215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.486182 6275 generic.go:356] "Generic (PLEG): container finished" podID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerID="f1dbe7cde9315b85cd36f5f7212efb4556586444d78aa711351c33fbd2ba6811" exitCode=0 Oct 30 02:36:53.486572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.486249 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" event={"ID":"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1","Type":"ContainerDied","Data":"f1dbe7cde9315b85cd36f5f7212efb4556586444d78aa711351c33fbd2ba6811"} Oct 30 02:36:53.656072 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.656014 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:36:53.725240 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.725206 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume\") pod \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " Oct 30 02:36:53.725450 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.725414 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb\" (UniqueName: \"kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb\") pod \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\" (UID: \"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1\") " Oct 30 02:36:53.725655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.725629 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume" pod "1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" (UID: "1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:53.726016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.725990 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:53.727660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.727625 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb" pod "1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" (UID: "1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1"). InnerVolumeSpecName "kube-api-access-v7wbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:53.827957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:53.827911 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wbb\" (UniqueName: \"kubernetes.io/projected/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1-kube-api-access-v7wbb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:36:54.496717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:54.496681 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" Oct 30 02:36:54.496717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:54.496695 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv" event={"ID":"1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1","Type":"ContainerDied","Data":"cbee08d158a7072dad7c5e645603477c76d9e259979d197b9363e71a24266bcb"} Oct 30 02:36:54.497279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:54.496758 6275 scope.go:122] "RemoveContainer" containerID="f1dbe7cde9315b85cd36f5f7212efb4556586444d78aa711351c33fbd2ba6811" Oct 30 02:36:54.558907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:54.558874 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv"] Oct 30 02:36:54.568795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:54.568766 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d207e8-5f6bf985f5-k6lcv"] Oct 30 02:36:55.782140 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:36:55.782087 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" path="/var/lib/kubelet/pods/1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1/volumes" Oct 30 02:37:09.844650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:09.844614 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn"] Oct 30 02:37:09.845078 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:09.844912 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" podUID="d0bd2b42-c8ae-4f03-aaec-766c64b870be" containerName="maas-api" containerID="cri-o://b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba" gracePeriod=30 Oct 30 02:37:10.228972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.228943 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:37:10.332148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.324717 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\") pod \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " Oct 30 02:37:10.332148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.324779 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26\" (UniqueName: \"kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26\") pod \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " Oct 30 02:37:10.332148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.325063 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs\") pod \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " Oct 30 02:37:10.332148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.325220 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca\") pod \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\" (UID: \"d0bd2b42-c8ae-4f03-aaec-766c64b870be\") " Oct 30 02:37:10.336145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.333321 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca" pod "d0bd2b42-c8ae-4f03-aaec-766c64b870be" (UID: "d0bd2b42-c8ae-4f03-aaec-766c64b870be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:37:10.354144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.349729 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26" pod "d0bd2b42-c8ae-4f03-aaec-766c64b870be" (UID: "d0bd2b42-c8ae-4f03-aaec-766c64b870be"). InnerVolumeSpecName "kube-api-access-9vh26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:10.354144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.349812 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls" pod "d0bd2b42-c8ae-4f03-aaec-766c64b870be" (UID: "d0bd2b42-c8ae-4f03-aaec-766c64b870be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:37:10.359792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.359737 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs" pod "d0bd2b42-c8ae-4f03-aaec-766c64b870be" (UID: "d0bd2b42-c8ae-4f03-aaec-766c64b870be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:37:10.428073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.428031 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:37:10.428073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.428072 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd2b42-c8ae-4f03-aaec-766c64b870be-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:37:10.428384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.428086 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bd2b42-c8ae-4f03-aaec-766c64b870be-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:37:10.428384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.428103 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vh26\" (UniqueName: \"kubernetes.io/projected/d0bd2b42-c8ae-4f03-aaec-766c64b870be-kube-api-access-9vh26\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:37:10.623193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.623151 6275 generic.go:356] "Generic (PLEG): container finished" podID="d0bd2b42-c8ae-4f03-aaec-766c64b870be" containerID="b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba" exitCode=0 Oct 30 02:37:10.623193 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.623186 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" event={"ID":"d0bd2b42-c8ae-4f03-aaec-766c64b870be","Type":"ContainerDied","Data":"b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba"} Oct 30 02:37:10.623440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.623208 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" Oct 30 02:37:10.623440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.623242 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn" event={"ID":"d0bd2b42-c8ae-4f03-aaec-766c64b870be","Type":"ContainerDied","Data":"64c87a27ff4f3613bf548193b4e05639aae1f869b08eb5c71652014aabb61887"} Oct 30 02:37:10.623440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.623271 6275 scope.go:122] "RemoveContainer" containerID="b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba" Oct 30 02:37:10.635655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.635627 6275 scope.go:122] "RemoveContainer" containerID="b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba" Oct 30 02:37:10.635952 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:37:10.635933 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba\": container with ID starting with b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba not found: ID does not exist" containerID="b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba" Oct 30 02:37:10.636005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.635961 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba"} err="failed to get container status \"b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba\": rpc error: code = NotFound desc = could not find container \"b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba\": container with ID starting with b3bbe7e1140ece26fa6f3be21b85964cf77af4d83314d45ced46584be25c64ba not found: ID does not exist" Oct 30 02:37:10.672809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.672767 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn"] Oct 30 02:37:10.682031 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:10.681994 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d207e8-59cd48fc58-pzwdn"] Oct 30 02:37:11.745633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:11.745601 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bd2b42-c8ae-4f03-aaec-766c64b870be" path="/var/lib/kubelet/pods/d0bd2b42-c8ae-4f03-aaec-766c64b870be/volumes" Oct 30 02:37:16.699292 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:37:16.699254 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:37:46.697581 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:37:46.697534 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:37:55.361876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.361840 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg"] Oct 30 02:37:55.362677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362660 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" Oct 30 02:37:55.362748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362678 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" Oct 30 02:37:55.362748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362688 6275 cpu_manager.go:414] "containerMap: removing container" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" containerName="payload-processing" Oct 30 02:37:55.362748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362693 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" containerName="payload-processing" Oct 30 02:37:55.362748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362735 6275 cpu_manager.go:414] "containerMap: removing container" podUID="61926654-587a-4e1c-b209-d0ef76657bca" containerName="maas-api" Oct 30 02:37:55.362748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362740 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61926654-587a-4e1c-b209-d0ef76657bca" containerName="maas-api" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362750 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerName="payload-pre-processing" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362756 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerName="payload-pre-processing" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362764 6275 cpu_manager.go:414] "containerMap: removing container" podUID="29f620af-6b5a-43d9-b385-4160459c791f" containerName="payload-pre-processing" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362769 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f620af-6b5a-43d9-b385-4160459c791f" containerName="payload-pre-processing" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362784 6275 cpu_manager.go:414] "containerMap: removing container" podUID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" containerName="main" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362789 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" containerName="main" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362799 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d0bd2b42-c8ae-4f03-aaec-766c64b870be" containerName="maas-api" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362804 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bd2b42-c8ae-4f03-aaec-766c64b870be" containerName="maas-api" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362907 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" Oct 30 02:37:55.362915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362918 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="61926654-587a-4e1c-b209-d0ef76657bca" containerName="maas-api" Oct 30 02:37:55.363472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362926 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7b7dbb-0fa2-43b1-a674-48a8a6f6c1d1" containerName="payload-pre-processing" Oct 30 02:37:55.363472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362932 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f620af-6b5a-43d9-b385-4160459c791f" containerName="payload-pre-processing" Oct 30 02:37:55.363472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362940 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="097a9ebe-4d14-49a7-ab68-cf45d7e66abc" containerName="main" Oct 30 02:37:55.363472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362947 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bd2b42-c8ae-4f03-aaec-766c64b870be" containerName="maas-api" Oct 30 02:37:55.363472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.362953 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb24450c-9ff5-45b0-98d8-721ae6b05aae" containerName="payload-processing" Oct 30 02:37:55.367005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.366990 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.378927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.378897 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.379199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.379171 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq74w\" (UniqueName: \"kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w\") pod \"payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.396278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.396243 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg"] Oct 30 02:37:55.396535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.396302 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-edae4317\"" Oct 30 02:37:55.499567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.499526 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.499991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.499968 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq74w\" (UniqueName: \"kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w\") pod \"payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.501571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.501544 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.559144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.555455 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-edae4317-dockercfg-s744k\"" Oct 30 02:37:55.644737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.644663 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq74w\" (UniqueName: \"kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w\") pod \"payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.652799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.652769 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts"] Oct 30 02:37:55.654116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.654092 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" Oct 30 02:37:55.654316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.654136 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" Oct 30 02:37:55.654393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.654362 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4ccd06-2cc3-4bad-baae-7e0993e5ccf7" containerName="istio-proxy" Oct 30 02:37:55.660777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.660753 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.666854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.666823 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts"] Oct 30 02:37:55.682619 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.682598 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:55.806511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.804899 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume\") pod \"payload-processing-e2e-mt-edae4317-69d497b588-dhfts\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.806511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.805303 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rb8v\" (UniqueName: \"kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v\") pod \"payload-processing-e2e-mt-edae4317-69d497b588-dhfts\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.906752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.906657 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rb8v\" (UniqueName: \"kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v\") pod \"payload-processing-e2e-mt-edae4317-69d497b588-dhfts\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.906936 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.906860 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume\") pod \"payload-processing-e2e-mt-edae4317-69d497b588-dhfts\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.907677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.907655 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume\") pod \"payload-processing-e2e-mt-edae4317-69d497b588-dhfts\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.938189 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.938158 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rb8v\" (UniqueName: \"kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v\") pod \"payload-processing-e2e-mt-edae4317-69d497b588-dhfts\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:55.982986 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:55.982957 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:56.020884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.020851 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg"] Oct 30 02:37:56.022708 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:37:56.022680 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc2bec3_8673_438e_9e66_ae67425803aa.slice/crio-42727ba8201097e0455c541cc67c7ee8e78721abb7b4855850da58053135f2eb WatchSource:0}: Error finding container 42727ba8201097e0455c541cc67c7ee8e78721abb7b4855850da58053135f2eb: Status 404 returned error can't find the container with id 42727ba8201097e0455c541cc67c7ee8e78721abb7b4855850da58053135f2eb Oct 30 02:37:56.301722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.301693 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts"] Oct 30 02:37:56.304466 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:37:56.304431 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fef1de_4941_4c89_91f4_dfb79be58160.slice/crio-7a49c299cf7c1fe9ce2753fad2db66456bbf69e69d8f06c053f5126674244335 WatchSource:0}: Error finding container 7a49c299cf7c1fe9ce2753fad2db66456bbf69e69d8f06c053f5126674244335: Status 404 returned error can't find the container with id 7a49c299cf7c1fe9ce2753fad2db66456bbf69e69d8f06c053f5126674244335 Oct 30 02:37:56.974290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.974254 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" event={"ID":"4dc2bec3-8673-438e-9e66-ae67425803aa","Type":"ContainerStarted","Data":"7a9d861a2299155301ac23bb630a670902c5f9d25c7c0f8c7f2398d888f62b18"} Oct 30 02:37:56.974746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.974336 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" event={"ID":"4dc2bec3-8673-438e-9e66-ae67425803aa","Type":"ContainerStarted","Data":"42727ba8201097e0455c541cc67c7ee8e78721abb7b4855850da58053135f2eb"} Oct 30 02:37:56.974746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.974369 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:37:56.975992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.975964 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" event={"ID":"10fef1de-4941-4c89-91f4-dfb79be58160","Type":"ContainerStarted","Data":"9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64"} Oct 30 02:37:56.976107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.976010 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" event={"ID":"10fef1de-4941-4c89-91f4-dfb79be58160","Type":"ContainerStarted","Data":"7a49c299cf7c1fe9ce2753fad2db66456bbf69e69d8f06c053f5126674244335"} Oct 30 02:37:56.976107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:56.976045 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:37:57.011276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:57.011230 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" podStartSLOduration=2.011214601 podStartE2EDuration="2.011214601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:37:57.010289279 +0000 UTC m=+4105.402096913" watchObservedRunningTime="2026-10-30 02:37:57.011214601 +0000 UTC m=+4105.403022235" Oct 30 02:37:57.051094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:37:57.051037 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" podStartSLOduration=2.0510178 podStartE2EDuration="2.0510178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:37:57.047850682 +0000 UTC m=+4105.439658314" watchObservedRunningTime="2026-10-30 02:37:57.0510178 +0000 UTC m=+4105.442825435" Oct 30 02:38:07.986221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:07.986190 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:38:07.986742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:07.986264 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:38:16.698354 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:38:16.698313 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:38:17.542176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:17.542139 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n"] Oct 30 02:38:22.542676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:22.542634 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerName="payload-processing" containerID="cri-o://29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b" gracePeriod=25 Oct 30 02:38:26.572910 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:26.572857 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:38:32.405105 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:32.405069 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf"] Oct 30 02:38:32.405857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:32.405824 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" containerID="cri-o://fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a" gracePeriod=30 Oct 30 02:38:32.844222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:32.844185 6275 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:38:32.844400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:32.844254 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:38:36.572309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:36.572264 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:38:37.691461 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.691399 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:38:37.819764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.819732 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.819972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.819779 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.819972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.819938 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.819972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.819963 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.820163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820006 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.820224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820149 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.820224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820187 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:38:37.820339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820269 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.820396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820357 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.820458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820418 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.820510 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820474 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data\") pod \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\" (UID: \"fc204e8d-6508-4956-9e7c-b9dafe5fa84e\") " Oct 30 02:38:37.820887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.820863 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.821068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.821042 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.821408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.821388 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.821506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.821414 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.821506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.821428 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.821506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.821431 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.821506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.821442 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.822333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.822297 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.822732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.822703 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:38:37.822817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.822800 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:37.822973 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.822953 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26" pod "fc204e8d-6508-4956-9e7c-b9dafe5fa84e" (UID: "fc204e8d-6508-4956-9e7c-b9dafe5fa84e"). InnerVolumeSpecName "kube-api-access-gjj26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:37.922516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.922486 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.922516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.922512 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.922516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.922522 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.922737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.922530 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj26\" (UniqueName: \"kubernetes.io/projected/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-kube-api-access-gjj26\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.922737 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:37.922539 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc204e8d-6508-4956-9e7c-b9dafe5fa84e-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:38.320819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.320785 6275 generic.go:356] "Generic (PLEG): container finished" podID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerID="fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a" exitCode=0 Oct 30 02:38:38.320985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.320844 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" Oct 30 02:38:38.320985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.320877 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" event={"ID":"fc204e8d-6508-4956-9e7c-b9dafe5fa84e","Type":"ContainerDied","Data":"fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a"} Oct 30 02:38:38.320985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.320933 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf" event={"ID":"fc204e8d-6508-4956-9e7c-b9dafe5fa84e","Type":"ContainerDied","Data":"d4499f9b626e3431d839160ae228c02ff4ef959be6e2e8b9508ecbce3a9f2c0b"} Oct 30 02:38:38.320985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.320954 6275 scope.go:122] "RemoveContainer" containerID="fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a" Oct 30 02:38:38.337087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.337056 6275 scope.go:122] "RemoveContainer" containerID="fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a" Oct 30 02:38:38.338516 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:38:38.338487 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a\": container with ID starting with fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a not found: ID does not exist" containerID="fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a" Oct 30 02:38:38.338638 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.338524 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a"} err="failed to get container status \"fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a\": rpc error: code = NotFound desc = could not find container \"fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a\": container with ID starting with fc683e8154fb3c08f211d2ac011ff5b59c8d02e14cba14fce91ab53db913e01a not found: ID does not exist" Oct 30 02:38:38.372460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.372422 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv"] Oct 30 02:38:38.373320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.373299 6275 cpu_manager.go:414] "containerMap: removing container" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" Oct 30 02:38:38.373320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.373318 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" Oct 30 02:38:38.373498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.373451 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" containerName="istio-proxy" Oct 30 02:38:38.378804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.378782 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.384593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.384569 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf"] Oct 30 02:38:38.389147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.388953 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pc5cf"] Oct 30 02:38:38.394837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.394811 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-edae4317-serving-cert\"" Oct 30 02:38:38.399081 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.399055 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv"] Oct 30 02:38:38.531111 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.531074 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.531316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.531237 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v468b\" (UniqueName: \"kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.531316 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.531270 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.531495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.531476 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.632977 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.632895 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.633153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.633028 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.633153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.633100 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v468b\" (UniqueName: \"kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.633260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.633164 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.633260 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:38:38.633211 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-serving-cert: secret "maas-api-e2e-mt-edae4317-serving-cert" not found Oct 30 02:38:38.633377 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:38:38.633285 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls podName:11837e56-a8e4-46f7-b265-09a108e54fa9 nodeName:}" failed. No retries permitted until 2026-10-30 02:38:39.133266287 +0000 UTC m=+4147.525073918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls") pod "maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" (UID: "11837e56-a8e4-46f7-b265-09a108e54fa9") : secret "maas-api-e2e-mt-edae4317-serving-cert" not found Oct 30 02:38:38.633654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.633631 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.635680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.635654 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:38.655828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:38.655801 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v468b\" (UniqueName: \"kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:39.097411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.097378 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp"] Oct 30 02:38:39.131699 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.131669 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt"] Oct 30 02:38:39.141046 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.141010 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:39.143607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.143583 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\") pod \"maas-api-e2e-mt-edae4317-6744969b7c-gkpcv\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:39.296130 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.296098 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:39.657156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.657112 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv"] Oct 30 02:38:39.659517 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:38:39.659484 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11837e56_a8e4_46f7_b265_09a108e54fa9.slice/crio-06b3a1bab32ece64fa75f1fe4c2b67d5872fb509a1c088767d628a3b64ca767e WatchSource:0}: Error finding container 06b3a1bab32ece64fa75f1fe4c2b67d5872fb509a1c088767d628a3b64ca767e: Status 404 returned error can't find the container with id 06b3a1bab32ece64fa75f1fe4c2b67d5872fb509a1c088767d628a3b64ca767e Oct 30 02:38:39.661165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.661111 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:38:39.741562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:39.741532 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc204e8d-6508-4956-9e7c-b9dafe5fa84e" path="/var/lib/kubelet/pods/fc204e8d-6508-4956-9e7c-b9dafe5fa84e/volumes" Oct 30 02:38:40.339686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:40.339599 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" event={"ID":"11837e56-a8e4-46f7-b265-09a108e54fa9","Type":"ContainerStarted","Data":"8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f"} Oct 30 02:38:40.339686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:40.339652 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" event={"ID":"11837e56-a8e4-46f7-b265-09a108e54fa9","Type":"ContainerStarted","Data":"06b3a1bab32ece64fa75f1fe4c2b67d5872fb509a1c088767d628a3b64ca767e"} Oct 30 02:38:40.340152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:40.339696 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:40.380549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:40.380474 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" podStartSLOduration=1.966482358 podStartE2EDuration="2.3804552s" totalImagesPullingTime="413.972842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:38:40.378351384 +0000 UTC m=+4148.770159035" watchObservedRunningTime="2026-10-30 02:38:40.3804552 +0000 UTC m=+4148.772262834" Oct 30 02:38:42.827543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:42.827516 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:38:42.917854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:42.917814 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh\") pod \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " Oct 30 02:38:42.918036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:42.917865 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume\") pod \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\" (UID: \"a52d08b9-a76e-4c95-a64d-3b2dbe73428d\") " Oct 30 02:38:42.918353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:42.918324 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume" pod "a52d08b9-a76e-4c95-a64d-3b2dbe73428d" (UID: "a52d08b9-a76e-4c95-a64d-3b2dbe73428d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:38:42.918547 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:42.918531 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:42.920080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:42.920053 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh" pod "a52d08b9-a76e-4c95-a64d-3b2dbe73428d" (UID: "a52d08b9-a76e-4c95-a64d-3b2dbe73428d"). InnerVolumeSpecName "kube-api-access-q4znh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:43.019494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.019466 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/a52d08b9-a76e-4c95-a64d-3b2dbe73428d-kube-api-access-q4znh\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:43.365613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.365579 6275 generic.go:356] "Generic (PLEG): container finished" podID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerID="29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b" exitCode=0 Oct 30 02:38:43.365613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.365617 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" event={"ID":"a52d08b9-a76e-4c95-a64d-3b2dbe73428d","Type":"ContainerDied","Data":"29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b"} Oct 30 02:38:43.365848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.365638 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" Oct 30 02:38:43.365848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.365662 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n" event={"ID":"a52d08b9-a76e-4c95-a64d-3b2dbe73428d","Type":"ContainerDied","Data":"905647571c475a38cc076d9008ed2acbe12435ad9e8561e11419a8c95f8ba9d2"} Oct 30 02:38:43.365848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.365684 6275 scope.go:122] "RemoveContainer" containerID="29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b" Oct 30 02:38:43.381878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.381858 6275 scope.go:122] "RemoveContainer" containerID="29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b" Oct 30 02:38:43.382174 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:38:43.382155 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b\": container with ID starting with 29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b not found: ID does not exist" containerID="29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b" Oct 30 02:38:43.382256 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.382182 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b"} err="failed to get container status \"29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b\": rpc error: code = NotFound desc = could not find container \"29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b\": container with ID starting with 29a2b5593deafe19d8b6c700007d23e331435bb917a6651c0aa31ace89be6c9b not found: ID does not exist" Oct 30 02:38:43.414616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.414586 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n"] Oct 30 02:38:43.424813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.424787 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52d131-64955cf7b8-5qx6n"] Oct 30 02:38:43.755647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:43.755565 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" path="/var/lib/kubelet/pods/a52d08b9-a76e-4c95-a64d-3b2dbe73428d/volumes" Oct 30 02:38:44.098098 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.098055 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" podUID="1155a223-6a0e-41f5-8966-0365df3ad93a" containerName="payload-processing" containerID="cri-o://6a84e021fc6e83080fc4ff92e98286c88241356640b623d1486701840622eb60" gracePeriod=25 Oct 30 02:38:44.132251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.132215 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerName="payload-pre-processing" containerID="cri-o://b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8" gracePeriod=25 Oct 30 02:38:44.373693 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.373665 6275 generic.go:356] "Generic (PLEG): container finished" podID="1155a223-6a0e-41f5-8966-0365df3ad93a" containerID="6a84e021fc6e83080fc4ff92e98286c88241356640b623d1486701840622eb60" exitCode=0 Oct 30 02:38:44.373844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.373734 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" event={"ID":"1155a223-6a0e-41f5-8966-0365df3ad93a","Type":"ContainerDied","Data":"6a84e021fc6e83080fc4ff92e98286c88241356640b623d1486701840622eb60"} Oct 30 02:38:44.373844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.373787 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" event={"ID":"1155a223-6a0e-41f5-8966-0365df3ad93a","Type":"ContainerDied","Data":"c9c7085c3cc25106cd20614c4dd99421df90e8008f8e014bee8555c4fa983ab6"} Oct 30 02:38:44.373844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.373804 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c7085c3cc25106cd20614c4dd99421df90e8008f8e014bee8555c4fa983ab6" Oct 30 02:38:44.390947 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.390925 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:38:44.449523 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.449491 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume\") pod \"1155a223-6a0e-41f5-8966-0365df3ad93a\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " Oct 30 02:38:44.449690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.449671 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x\" (UniqueName: \"kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x\") pod \"1155a223-6a0e-41f5-8966-0365df3ad93a\" (UID: \"1155a223-6a0e-41f5-8966-0365df3ad93a\") " Oct 30 02:38:44.449919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.449897 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume" pod "1155a223-6a0e-41f5-8966-0365df3ad93a" (UID: "1155a223-6a0e-41f5-8966-0365df3ad93a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:38:44.450252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.450230 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1155a223-6a0e-41f5-8966-0365df3ad93a-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:44.451886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.451856 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x" pod "1155a223-6a0e-41f5-8966-0365df3ad93a" (UID: "1155a223-6a0e-41f5-8966-0365df3ad93a"). InnerVolumeSpecName "kube-api-access-cql2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:44.551998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:44.551961 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cql2x\" (UniqueName: \"kubernetes.io/projected/1155a223-6a0e-41f5-8966-0365df3ad93a-kube-api-access-cql2x\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:38:45.383652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:45.383618 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp" Oct 30 02:38:45.431363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:45.431332 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp"] Oct 30 02:38:45.441251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:45.441221 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07fb25-5c59fb4c8b-rcfwp"] Oct 30 02:38:45.742588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:45.742515 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1155a223-6a0e-41f5-8966-0365df3ad93a" path="/var/lib/kubelet/pods/1155a223-6a0e-41f5-8966-0365df3ad93a/volumes" Oct 30 02:38:46.355289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:46.355263 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:38:46.698725 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:38:46.698644 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:38:52.164364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:38:52.164321 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:39:02.163996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:02.163952 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:39:04.407874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.407812 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:39:04.477895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.477852 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt\" (UniqueName: \"kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt\") pod \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " Oct 30 02:39:04.478185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.478164 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume\") pod \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\" (UID: \"8515cfeb-db30-420b-95f3-7f84e37f1e3a\") " Oct 30 02:39:04.478633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.478599 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume" pod "8515cfeb-db30-420b-95f3-7f84e37f1e3a" (UID: "8515cfeb-db30-420b-95f3-7f84e37f1e3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:04.478891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.478869 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515cfeb-db30-420b-95f3-7f84e37f1e3a-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:04.480190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.480160 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt" pod "8515cfeb-db30-420b-95f3-7f84e37f1e3a" (UID: "8515cfeb-db30-420b-95f3-7f84e37f1e3a"). InnerVolumeSpecName "kube-api-access-sqvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:04.532283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.532247 6275 generic.go:356] "Generic (PLEG): container finished" podID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerID="b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8" exitCode=0 Oct 30 02:39:04.532441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.532294 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" Oct 30 02:39:04.532441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.532336 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" event={"ID":"8515cfeb-db30-420b-95f3-7f84e37f1e3a","Type":"ContainerDied","Data":"b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8"} Oct 30 02:39:04.532441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.532389 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt" event={"ID":"8515cfeb-db30-420b-95f3-7f84e37f1e3a","Type":"ContainerDied","Data":"bc961197e3608afc52f799929dc79b7822250101ed0ea1107700da716b70032e"} Oct 30 02:39:04.532441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.532416 6275 scope.go:122] "RemoveContainer" containerID="b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8" Oct 30 02:39:04.543711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.543692 6275 scope.go:122] "RemoveContainer" containerID="b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8" Oct 30 02:39:04.543974 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:04.543954 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8\": container with ID starting with b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8 not found: ID does not exist" containerID="b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8" Oct 30 02:39:04.544022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.543982 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8"} err="failed to get container status \"b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8\": rpc error: code = NotFound desc = could not find container \"b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8\": container with ID starting with b80e6cac3d771cb16c61b558f63c9aec3bdffa4d8287d600b70bffe7d714f2d8 not found: ID does not exist" Oct 30 02:39:04.580015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.579975 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt"] Oct 30 02:39:04.580211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.580060 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvpt\" (UniqueName: \"kubernetes.io/projected/8515cfeb-db30-420b-95f3-7f84e37f1e3a-kube-api-access-sqvpt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:04.584704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:04.584681 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07fb25-654579f6bc-gj6qt"] Oct 30 02:39:05.741506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:05.741474 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" path="/var/lib/kubelet/pods/8515cfeb-db30-420b-95f3-7f84e37f1e3a/volumes" Oct 30 02:39:16.704781 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:16.704743 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:39:16.935051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:16.935014 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw"] Oct 30 02:39:16.975373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:16.975291 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64"] Oct 30 02:39:17.581744 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:17.581703 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf"] Oct 30 02:39:17.581991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:17.581961 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" podUID="ea512869-be10-4520-a3e1-6d1827de552f" containerName="maas-api" containerID="cri-o://03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52" gracePeriod=30 Oct 30 02:39:18.084142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.076288 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:39:18.235175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.234707 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv\" (UniqueName: \"kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv\") pod \"ea512869-be10-4520-a3e1-6d1827de552f\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " Oct 30 02:39:18.235175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.234799 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca\") pod \"ea512869-be10-4520-a3e1-6d1827de552f\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " Oct 30 02:39:18.235466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.235280 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca" pod "ea512869-be10-4520-a3e1-6d1827de552f" (UID: "ea512869-be10-4520-a3e1-6d1827de552f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:18.235466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.235416 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls\") pod \"ea512869-be10-4520-a3e1-6d1827de552f\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " Oct 30 02:39:18.235749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.235723 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs\") pod \"ea512869-be10-4520-a3e1-6d1827de552f\" (UID: \"ea512869-be10-4520-a3e1-6d1827de552f\") " Oct 30 02:39:18.236730 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.236691 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea512869-be10-4520-a3e1-6d1827de552f-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:18.237642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.237618 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv" pod "ea512869-be10-4520-a3e1-6d1827de552f" (UID: "ea512869-be10-4520-a3e1-6d1827de552f"). InnerVolumeSpecName "kube-api-access-8wzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:18.238908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.238884 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls" pod "ea512869-be10-4520-a3e1-6d1827de552f" (UID: "ea512869-be10-4520-a3e1-6d1827de552f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:18.238908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.238894 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs" pod "ea512869-be10-4520-a3e1-6d1827de552f" (UID: "ea512869-be10-4520-a3e1-6d1827de552f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:18.338051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.338017 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:18.338051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.338050 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzgv\" (UniqueName: \"kubernetes.io/projected/ea512869-be10-4520-a3e1-6d1827de552f-kube-api-access-8wzgv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:18.338345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.338060 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea512869-be10-4520-a3e1-6d1827de552f-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:18.639895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.639831 6275 generic.go:356] "Generic (PLEG): container finished" podID="ea512869-be10-4520-a3e1-6d1827de552f" containerID="03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52" exitCode=0 Oct 30 02:39:18.639895 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.639884 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" Oct 30 02:39:18.640117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.639929 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" event={"ID":"ea512869-be10-4520-a3e1-6d1827de552f","Type":"ContainerDied","Data":"03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52"} Oct 30 02:39:18.640117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.639999 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf" event={"ID":"ea512869-be10-4520-a3e1-6d1827de552f","Type":"ContainerDied","Data":"31ba927c2a4d736187de7127986688bb4ffe5de0b78e1b3a6325c2a903963fbf"} Oct 30 02:39:18.640117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.640027 6275 scope.go:122] "RemoveContainer" containerID="03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52" Oct 30 02:39:18.651507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.651490 6275 scope.go:122] "RemoveContainer" containerID="03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52" Oct 30 02:39:18.651763 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:18.651747 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52\": container with ID starting with 03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52 not found: ID does not exist" containerID="03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52" Oct 30 02:39:18.651808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.651773 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52"} err="failed to get container status \"03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52\": rpc error: code = NotFound desc = could not find container \"03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52\": container with ID starting with 03c7bb21bd11e1e95aa763545a6cb3ad0df15e59b8ec0de8d0b325e1e301ef52 not found: ID does not exist" Oct 30 02:39:18.688295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.688261 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf"] Oct 30 02:39:18.700957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:18.700903 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07fb25-567ff97b7b-pndwf"] Oct 30 02:39:19.742371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:19.742334 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea512869-be10-4520-a3e1-6d1827de552f" path="/var/lib/kubelet/pods/ea512869-be10-4520-a3e1-6d1827de552f/volumes" Oct 30 02:39:21.937503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:21.937202 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerName="payload-processing" containerID="cri-o://8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1" gracePeriod=25 Oct 30 02:39:21.976300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:21.976238 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerName="payload-pre-processing" containerID="cri-o://f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab" gracePeriod=25 Oct 30 02:39:24.668935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:24.668895 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:39:24.669400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:24.668945 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:39:32.633067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:32.633036 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:39:32.640827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:32.640806 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:39:34.668429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:34.668389 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:39:34.668869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:34.668392 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:39:35.807201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:35.807170 6275 scope.go:122] "RemoveContainer" containerID="6a84e021fc6e83080fc4ff92e98286c88241356640b623d1486701840622eb60" Oct 30 02:39:38.332615 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.332573 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr"] Oct 30 02:39:38.333011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.332875 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" podUID="428a6289-6cc8-4c54-8103-9833110bb7c3" containerName="maas-api" containerID="cri-o://04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242" gracePeriod=30 Oct 30 02:39:38.752150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.749372 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:39:38.772992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.772955 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca\") pod \"428a6289-6cc8-4c54-8103-9833110bb7c3\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " Oct 30 02:39:38.773475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.773447 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs\") pod \"428a6289-6cc8-4c54-8103-9833110bb7c3\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " Oct 30 02:39:38.774609 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.774585 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m\" (UniqueName: \"kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m\") pod \"428a6289-6cc8-4c54-8103-9833110bb7c3\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " Oct 30 02:39:38.774871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.774851 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls\") pod \"428a6289-6cc8-4c54-8103-9833110bb7c3\" (UID: \"428a6289-6cc8-4c54-8103-9833110bb7c3\") " Oct 30 02:39:38.780314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.776540 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca" pod "428a6289-6cc8-4c54-8103-9833110bb7c3" (UID: "428a6289-6cc8-4c54-8103-9833110bb7c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:38.788065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.788037 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls" pod "428a6289-6cc8-4c54-8103-9833110bb7c3" (UID: "428a6289-6cc8-4c54-8103-9833110bb7c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:38.825503 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.825419 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m" pod "428a6289-6cc8-4c54-8103-9833110bb7c3" (UID: "428a6289-6cc8-4c54-8103-9833110bb7c3"). InnerVolumeSpecName "kube-api-access-xgt5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:38.834148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.831269 6275 generic.go:356] "Generic (PLEG): container finished" podID="428a6289-6cc8-4c54-8103-9833110bb7c3" containerID="04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242" exitCode=0 Oct 30 02:39:38.834148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.831522 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" event={"ID":"428a6289-6cc8-4c54-8103-9833110bb7c3","Type":"ContainerDied","Data":"04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242"} Oct 30 02:39:38.834148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.831569 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" event={"ID":"428a6289-6cc8-4c54-8103-9833110bb7c3","Type":"ContainerDied","Data":"31fa19956d3a1050af39c4a333f40367b42e9fd9d24ac65c7dfc2a23f39032be"} Oct 30 02:39:38.834148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.831597 6275 scope.go:122] "RemoveContainer" containerID="04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242" Oct 30 02:39:38.834148 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.831753 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr" Oct 30 02:39:38.834618 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.834583 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs" pod "428a6289-6cc8-4c54-8103-9833110bb7c3" (UID: "428a6289-6cc8-4c54-8103-9833110bb7c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:38.871064 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.871027 6275 scope.go:122] "RemoveContainer" containerID="04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242" Oct 30 02:39:38.873823 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:38.873701 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242\": container with ID starting with 04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242 not found: ID does not exist" containerID="04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242" Oct 30 02:39:38.873823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.873749 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242"} err="failed to get container status \"04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242\": rpc error: code = NotFound desc = could not find container \"04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242\": container with ID starting with 04c5e2fa3f4490b20797a1df5eff30e3ce5656a512d72c53f4e6b0e6413c1242 not found: ID does not exist" Oct 30 02:39:38.878159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.878078 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:38.878159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.878110 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgt5m\" (UniqueName: \"kubernetes.io/projected/428a6289-6cc8-4c54-8103-9833110bb7c3-kube-api-access-xgt5m\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:38.878410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.878371 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428a6289-6cc8-4c54-8103-9833110bb7c3-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:38.878410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:38.878393 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428a6289-6cc8-4c54-8103-9833110bb7c3-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:39.205167 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:39.203602 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr"] Oct 30 02:39:39.207584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:39.207554 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58fad4-7757f5df5c-dsjdr"] Oct 30 02:39:39.744112 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:39.744076 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428a6289-6cc8-4c54-8103-9833110bb7c3" path="/var/lib/kubelet/pods/428a6289-6cc8-4c54-8103-9833110bb7c3/volumes" Oct 30 02:39:42.271967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.271931 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:39:42.324357 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.324314 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:39:42.339058 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.339023 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf\" (UniqueName: \"kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf\") pod \"7eae7013-7f1b-4066-babf-ed2ee657212a\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " Oct 30 02:39:42.339297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.339276 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume\") pod \"7eae7013-7f1b-4066-babf-ed2ee657212a\" (UID: \"7eae7013-7f1b-4066-babf-ed2ee657212a\") " Oct 30 02:39:42.339862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.339827 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume" pod "7eae7013-7f1b-4066-babf-ed2ee657212a" (UID: "7eae7013-7f1b-4066-babf-ed2ee657212a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:42.341386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.341362 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf" pod "7eae7013-7f1b-4066-babf-ed2ee657212a" (UID: "7eae7013-7f1b-4066-babf-ed2ee657212a"). InnerVolumeSpecName "kube-api-access-wv4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:42.445175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.444669 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl\" (UniqueName: \"kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl\") pod \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " Oct 30 02:39:42.445373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.445170 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume\") pod \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\" (UID: \"1b82dfb5-2853-4663-8c19-c83c5d32ec0b\") " Oct 30 02:39:42.446263 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.445974 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eae7013-7f1b-4066-babf-ed2ee657212a-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:42.446263 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.446082 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv4jf\" (UniqueName: \"kubernetes.io/projected/7eae7013-7f1b-4066-babf-ed2ee657212a-kube-api-access-wv4jf\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:42.455773 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.455733 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume" pod "1b82dfb5-2853-4663-8c19-c83c5d32ec0b" (UID: "1b82dfb5-2853-4663-8c19-c83c5d32ec0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:42.458423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.458397 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl" pod "1b82dfb5-2853-4663-8c19-c83c5d32ec0b" (UID: "1b82dfb5-2853-4663-8c19-c83c5d32ec0b"). InnerVolumeSpecName "kube-api-access-rcrrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:42.547186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.547153 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:42.547186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.547177 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrrl\" (UniqueName: \"kubernetes.io/projected/1b82dfb5-2853-4663-8c19-c83c5d32ec0b-kube-api-access-rcrrl\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:42.860669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.860589 6275 generic.go:356] "Generic (PLEG): container finished" podID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerID="8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1" exitCode=0 Oct 30 02:39:42.860669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.860636 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" Oct 30 02:39:42.860885 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.860668 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" event={"ID":"7eae7013-7f1b-4066-babf-ed2ee657212a","Type":"ContainerDied","Data":"8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1"} Oct 30 02:39:42.860885 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.860726 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw" event={"ID":"7eae7013-7f1b-4066-babf-ed2ee657212a","Type":"ContainerDied","Data":"b6ae8319f2a1d7e117cf16d14e0965efe5b261ee3139c76bef721ba3fdc0651b"} Oct 30 02:39:42.860885 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.860755 6275 scope.go:122] "RemoveContainer" containerID="8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1" Oct 30 02:39:42.862340 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.862314 6275 generic.go:356] "Generic (PLEG): container finished" podID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerID="f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab" exitCode=0 Oct 30 02:39:42.862488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.862371 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" Oct 30 02:39:42.862488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.862398 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" event={"ID":"1b82dfb5-2853-4663-8c19-c83c5d32ec0b","Type":"ContainerDied","Data":"f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab"} Oct 30 02:39:42.862488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.862447 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64" event={"ID":"1b82dfb5-2853-4663-8c19-c83c5d32ec0b","Type":"ContainerDied","Data":"23bb851ea3661efeca11dcda7450864a970d5b8f633fc214c7db2fe1dfca176f"} Oct 30 02:39:42.872094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.872061 6275 scope.go:122] "RemoveContainer" containerID="8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1" Oct 30 02:39:42.872495 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:42.872393 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1\": container with ID starting with 8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1 not found: ID does not exist" containerID="8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1" Oct 30 02:39:42.872495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.872428 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1"} err="failed to get container status \"8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1\": rpc error: code = NotFound desc = could not find container \"8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1\": container with ID starting with 8f3801969cba2189f9714a150bcb7517d0952c08149715608a9a6315f77f7fb1 not found: ID does not exist" Oct 30 02:39:42.872495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.872453 6275 scope.go:122] "RemoveContainer" containerID="f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab" Oct 30 02:39:42.882865 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.882849 6275 scope.go:122] "RemoveContainer" containerID="f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab" Oct 30 02:39:42.883218 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:42.883191 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab\": container with ID starting with f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab not found: ID does not exist" containerID="f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab" Oct 30 02:39:42.883288 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.883221 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab"} err="failed to get container status \"f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab\": rpc error: code = NotFound desc = could not find container \"f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab\": container with ID starting with f1df694374987338502c70865ed312662177aa03892168478046d3dfb28310ab not found: ID does not exist" Oct 30 02:39:42.920825 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.920794 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw"] Oct 30 02:39:42.930825 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.930795 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58fad4-75547c5dfb-kp4hw"] Oct 30 02:39:42.975306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.975199 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64"] Oct 30 02:39:42.978680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:42.978651 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58fad4-5868577ff9-wkf64"] Oct 30 02:39:43.741505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:43.741471 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" path="/var/lib/kubelet/pods/1b82dfb5-2853-4663-8c19-c83c5d32ec0b/volumes" Oct 30 02:39:43.742086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:43.742065 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" path="/var/lib/kubelet/pods/7eae7013-7f1b-4066-babf-ed2ee657212a/volumes" Oct 30 02:39:46.697891 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:46.697854 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:39:49.244741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:49.244704 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs"] Oct 30 02:39:53.882097 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:53.882061 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl"] Oct 30 02:39:53.882591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:53.882299 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" podUID="600da90f-44d7-45ec-8dab-41133df0f8cc" containerName="maas-api" containerID="cri-o://96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f" gracePeriod=30 Oct 30 02:39:54.228338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.228310 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:39:54.245455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.245394 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerName="payload-pre-processing" containerID="cri-o://1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719" gracePeriod=25 Oct 30 02:39:54.289704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.282804 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca\") pod \"600da90f-44d7-45ec-8dab-41133df0f8cc\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " Oct 30 02:39:54.289704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.282869 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs\" (UniqueName: \"kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs\") pod \"600da90f-44d7-45ec-8dab-41133df0f8cc\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " Oct 30 02:39:54.289704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.282959 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs\") pod \"600da90f-44d7-45ec-8dab-41133df0f8cc\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " Oct 30 02:39:54.289704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.283026 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls\") pod \"600da90f-44d7-45ec-8dab-41133df0f8cc\" (UID: \"600da90f-44d7-45ec-8dab-41133df0f8cc\") " Oct 30 02:39:54.309147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.305777 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca" pod "600da90f-44d7-45ec-8dab-41133df0f8cc" (UID: "600da90f-44d7-45ec-8dab-41133df0f8cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:54.322408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.315656 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs" pod "600da90f-44d7-45ec-8dab-41133df0f8cc" (UID: "600da90f-44d7-45ec-8dab-41133df0f8cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:54.361962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.359242 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls" pod "600da90f-44d7-45ec-8dab-41133df0f8cc" (UID: "600da90f-44d7-45ec-8dab-41133df0f8cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:54.361962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.360552 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs" pod "600da90f-44d7-45ec-8dab-41133df0f8cc" (UID: "600da90f-44d7-45ec-8dab-41133df0f8cc"). InnerVolumeSpecName "kube-api-access-8cmhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:54.406456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.406417 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:54.406456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.406458 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600da90f-44d7-45ec-8dab-41133df0f8cc-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:54.406747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.406474 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600da90f-44d7-45ec-8dab-41133df0f8cc-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:54.406747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.406490 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cmhs\" (UniqueName: \"kubernetes.io/projected/600da90f-44d7-45ec-8dab-41133df0f8cc-kube-api-access-8cmhs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:39:54.958021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.957984 6275 generic.go:356] "Generic (PLEG): container finished" podID="600da90f-44d7-45ec-8dab-41133df0f8cc" containerID="96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f" exitCode=0 Oct 30 02:39:54.958464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.958032 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" Oct 30 02:39:54.958464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.958066 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" event={"ID":"600da90f-44d7-45ec-8dab-41133df0f8cc","Type":"ContainerDied","Data":"96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f"} Oct 30 02:39:54.958464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.958117 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl" event={"ID":"600da90f-44d7-45ec-8dab-41133df0f8cc","Type":"ContainerDied","Data":"fa53fcc5a559d8284452562bb2a94de7dbf85df7df5b7c2a189e9af24dc5c10c"} Oct 30 02:39:54.958464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.958155 6275 scope.go:122] "RemoveContainer" containerID="96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f" Oct 30 02:39:54.968801 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.968783 6275 scope.go:122] "RemoveContainer" containerID="96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f" Oct 30 02:39:54.969098 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:39:54.969081 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f\": container with ID starting with 96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f not found: ID does not exist" containerID="96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f" Oct 30 02:39:54.969170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:54.969106 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f"} err="failed to get container status \"96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f\": rpc error: code = NotFound desc = could not find container \"96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f\": container with ID starting with 96f5f135152151d28a92528a9c7813a3f5eda236f5c4f79ebf7702de3e517c3f not found: ID does not exist" Oct 30 02:39:55.005960 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:55.005891 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl"] Oct 30 02:39:55.015818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:55.015797 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-93ec46-d47cb7c56-xpqkl"] Oct 30 02:39:55.740858 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:39:55.740826 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600da90f-44d7-45ec-8dab-41133df0f8cc" path="/var/lib/kubelet/pods/600da90f-44d7-45ec-8dab-41133df0f8cc/volumes" Oct 30 02:40:02.397158 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:02.397099 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:12.397541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:12.397455 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:14.523747 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.523719 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:40:14.613741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.613697 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47\" (UniqueName: \"kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47\") pod \"4b095cab-6798-4fcc-960d-fdf54d7e6522\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " Oct 30 02:40:14.613988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.613970 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume\") pod \"4b095cab-6798-4fcc-960d-fdf54d7e6522\" (UID: \"4b095cab-6798-4fcc-960d-fdf54d7e6522\") " Oct 30 02:40:14.614395 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.614367 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume" pod "4b095cab-6798-4fcc-960d-fdf54d7e6522" (UID: "4b095cab-6798-4fcc-960d-fdf54d7e6522"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:14.614634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.614607 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b095cab-6798-4fcc-960d-fdf54d7e6522-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:14.614938 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.614916 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn"] Oct 30 02:40:14.615170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.615143 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" containerID="cri-o://6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70" gracePeriod=30 Oct 30 02:40:14.616690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.616668 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47" pod "4b095cab-6798-4fcc-960d-fdf54d7e6522" (UID: "4b095cab-6798-4fcc-960d-fdf54d7e6522"). InnerVolumeSpecName "kube-api-access-clm47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:14.716612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:14.716572 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clm47\" (UniqueName: \"kubernetes.io/projected/4b095cab-6798-4fcc-960d-fdf54d7e6522-kube-api-access-clm47\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:15.162533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.162496 6275 generic.go:356] "Generic (PLEG): container finished" podID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerID="1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719" exitCode=0 Oct 30 02:40:15.162727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.162549 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" event={"ID":"4b095cab-6798-4fcc-960d-fdf54d7e6522","Type":"ContainerDied","Data":"1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719"} Oct 30 02:40:15.162727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.162564 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" Oct 30 02:40:15.162727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.162601 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs" event={"ID":"4b095cab-6798-4fcc-960d-fdf54d7e6522","Type":"ContainerDied","Data":"c5b5d02dccce137af122201eb2165be78aaa7179baf57cd9bd5ee40fc2dedba6"} Oct 30 02:40:15.162727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.162629 6275 scope.go:122] "RemoveContainer" containerID="1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719" Oct 30 02:40:15.175084 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.175057 6275 scope.go:122] "RemoveContainer" containerID="1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719" Oct 30 02:40:15.175457 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:15.175422 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719\": container with ID starting with 1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719 not found: ID does not exist" containerID="1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719" Oct 30 02:40:15.175564 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.175467 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719"} err="failed to get container status \"1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719\": rpc error: code = NotFound desc = could not find container \"1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719\": container with ID starting with 1c3e7db5e452f44ad691de213b03a91cbb4b2cd595853224498973dc3ad90719 not found: ID does not exist" Oct 30 02:40:15.212637 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.212600 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs"] Oct 30 02:40:15.223009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.222974 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-93ec46-6d87dc7b8c-p7gfs"] Oct 30 02:40:15.778982 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:15.778947 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" path="/var/lib/kubelet/pods/4b095cab-6798-4fcc-960d-fdf54d7e6522/volumes" Oct 30 02:40:16.775146 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:16.767310 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:40:19.912413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.912379 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 02:40:19.965538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.965500 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.965827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.965794 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:19.965996 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.965968 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966044 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966104 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966255 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966291 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966305 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:19.966424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966353 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966451 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966497 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo\") pod \"ae5421aa-5367-446c-b3f2-8261bf43d585\" (UID: \"ae5421aa-5367-446c-b3f2-8261bf43d585\") " Oct 30 02:40:19.966659 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966630 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:19.967317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.966974 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:19.967317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.967158 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:19.967317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.967181 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:19.967317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.967196 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5421aa-5367-446c-b3f2-8261bf43d585-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:19.967317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.967208 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:19.967317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.967276 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:19.969012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.968986 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:19.969338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.969307 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:19.969446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.969348 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "kube-api-access-6zwgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:19.969446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:19.969427 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo" pod "ae5421aa-5367-446c-b3f2-8261bf43d585" (UID: "ae5421aa-5367-446c-b3f2-8261bf43d585"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:40:20.069542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.069505 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:20.069542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.069545 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:20.069768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.069560 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:20.069768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.069574 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zwgh\" (UniqueName: \"kubernetes.io/projected/ae5421aa-5367-446c-b3f2-8261bf43d585-kube-api-access-6zwgh\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:20.069768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.069588 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5421aa-5367-446c-b3f2-8261bf43d585-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:20.247512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.247429 6275 generic.go:356] "Generic (PLEG): container finished" podID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerID="6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70" exitCode=0 Oct 30 02:40:20.247512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.247471 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" event={"ID":"ae5421aa-5367-446c-b3f2-8261bf43d585","Type":"ContainerDied","Data":"6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70"} Oct 30 02:40:20.247512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.247477 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" Oct 30 02:40:20.247512 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.247510 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn" event={"ID":"ae5421aa-5367-446c-b3f2-8261bf43d585","Type":"ContainerDied","Data":"578cb744632f68a7a1fb4f0b4e3513117ead8d26ee0c3f6906b4736039ce7d6d"} Oct 30 02:40:20.247920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.247531 6275 scope.go:122] "RemoveContainer" containerID="6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70" Oct 30 02:40:20.259736 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.259713 6275 scope.go:122] "RemoveContainer" containerID="6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70" Oct 30 02:40:20.260006 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:20.259984 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70\": container with ID starting with 6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70 not found: ID does not exist" containerID="6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70" Oct 30 02:40:20.260081 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.260015 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70"} err="failed to get container status \"6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70\": rpc error: code = NotFound desc = could not find container \"6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70\": container with ID starting with 6afac326c58434cc1e02e498e34c581144d059283aa8b8faeca45dbf00ed6c70 not found: ID does not exist" Oct 30 02:40:20.296099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.296066 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn"] Oct 30 02:40:20.306133 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:20.306086 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-qw6sn"] Oct 30 02:40:21.755142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:21.753949 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" path="/var/lib/kubelet/pods/ae5421aa-5367-446c-b3f2-8261bf43d585/volumes" Oct 30 02:40:25.402117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.402078 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8"] Oct 30 02:40:25.402648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.402376 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" podUID="ae9b6424-2631-48fb-a12c-42dd5303a752" containerName="maas-api" containerID="cri-o://4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c" gracePeriod=30 Oct 30 02:40:25.745370 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.745343 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:40:25.893479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.893210 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls\") pod \"ae9b6424-2631-48fb-a12c-42dd5303a752\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " Oct 30 02:40:25.893479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.893278 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g\" (UniqueName: \"kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g\") pod \"ae9b6424-2631-48fb-a12c-42dd5303a752\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " Oct 30 02:40:25.893479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.893385 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca\") pod \"ae9b6424-2631-48fb-a12c-42dd5303a752\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " Oct 30 02:40:25.893808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.893610 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs\") pod \"ae9b6424-2631-48fb-a12c-42dd5303a752\" (UID: \"ae9b6424-2631-48fb-a12c-42dd5303a752\") " Oct 30 02:40:25.893808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.893756 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca" pod "ae9b6424-2631-48fb-a12c-42dd5303a752" (UID: "ae9b6424-2631-48fb-a12c-42dd5303a752"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:25.894309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.894279 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9b6424-2631-48fb-a12c-42dd5303a752-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:25.895705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.895674 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls" pod "ae9b6424-2631-48fb-a12c-42dd5303a752" (UID: "ae9b6424-2631-48fb-a12c-42dd5303a752"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:40:25.895951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.895926 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g" pod "ae9b6424-2631-48fb-a12c-42dd5303a752" (UID: "ae9b6424-2631-48fb-a12c-42dd5303a752"). InnerVolumeSpecName "kube-api-access-l4j2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:25.896364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.896339 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs" pod "ae9b6424-2631-48fb-a12c-42dd5303a752" (UID: "ae9b6424-2631-48fb-a12c-42dd5303a752"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:40:25.995856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.995765 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:25.995856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.995800 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae9b6424-2631-48fb-a12c-42dd5303a752-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:25.995856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:25.995811 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4j2g\" (UniqueName: \"kubernetes.io/projected/ae9b6424-2631-48fb-a12c-42dd5303a752-kube-api-access-l4j2g\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:26.295244 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.295206 6275 generic.go:356] "Generic (PLEG): container finished" podID="ae9b6424-2631-48fb-a12c-42dd5303a752" containerID="4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c" exitCode=0 Oct 30 02:40:26.295498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.295239 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" event={"ID":"ae9b6424-2631-48fb-a12c-42dd5303a752","Type":"ContainerDied","Data":"4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c"} Oct 30 02:40:26.295498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.295254 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" Oct 30 02:40:26.295498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.295298 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8" event={"ID":"ae9b6424-2631-48fb-a12c-42dd5303a752","Type":"ContainerDied","Data":"0bcfa960248a43a3a5d04fd985a265c6c1e718eee5188002ef0e9c623cbf536b"} Oct 30 02:40:26.295498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.295325 6275 scope.go:122] "RemoveContainer" containerID="4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c" Oct 30 02:40:26.306799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.306778 6275 scope.go:122] "RemoveContainer" containerID="4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c" Oct 30 02:40:26.307090 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:26.307073 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c\": container with ID starting with 4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c not found: ID does not exist" containerID="4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c" Oct 30 02:40:26.307208 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.307098 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c"} err="failed to get container status \"4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c\": rpc error: code = NotFound desc = could not find container \"4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c\": container with ID starting with 4ad47797925c35d47094f21e8b7063a7d61ba90b5cc9dc421700e1e6c9ea3e2c not found: ID does not exist" Oct 30 02:40:26.344166 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.344114 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8"] Oct 30 02:40:26.354963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:26.354931 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4153e5-6dd88f7498-7pgp8"] Oct 30 02:40:27.744704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:27.744671 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9b6424-2631-48fb-a12c-42dd5303a752" path="/var/lib/kubelet/pods/ae9b6424-2631-48fb-a12c-42dd5303a752/volumes" Oct 30 02:40:33.649156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:33.649103 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn"] Oct 30 02:40:33.649685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:33.649396 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" containerID="cri-o://e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c" gracePeriod=30 Oct 30 02:40:33.799295 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:33.799258 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:40:33.799713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:33.799678 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:40:38.934319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.934248 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:40:38.980447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980415 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980482 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980567 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980628 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980650 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980704 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980718 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:38.980772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980754 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980968 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980828 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.980968 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980894 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:38.981137 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981001 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:38.981230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981087 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:38.981230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981099 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:38.981230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.980897 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data\") pod \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\" (UID: \"68d1c41c-7055-41c5-9a20-3a1b41dd709b\") " Oct 30 02:40:38.981718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981695 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:38.981718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981715 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:38.981906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981725 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:38.981906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981734 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:38.981906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.981744 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:38.983037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.983009 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:40:38.983150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.983086 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "kube-api-access-s5bjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:38.983276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.983256 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:38.983318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:38.983274 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token" pod "68d1c41c-7055-41c5-9a20-3a1b41dd709b" (UID: "68d1c41c-7055-41c5-9a20-3a1b41dd709b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:39.083487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.083451 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:39.083487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.083480 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:39.083487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.083489 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bjc\" (UniqueName: \"kubernetes.io/projected/68d1c41c-7055-41c5-9a20-3a1b41dd709b-kube-api-access-s5bjc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:39.083729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.083500 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d1c41c-7055-41c5-9a20-3a1b41dd709b-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:40:39.396475 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.396443 6275 generic.go:356] "Generic (PLEG): container finished" podID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerID="e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c" exitCode=0 Oct 30 02:40:39.396689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.396502 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" Oct 30 02:40:39.396689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.396522 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" event={"ID":"68d1c41c-7055-41c5-9a20-3a1b41dd709b","Type":"ContainerDied","Data":"e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c"} Oct 30 02:40:39.396689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.396573 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn" event={"ID":"68d1c41c-7055-41c5-9a20-3a1b41dd709b","Type":"ContainerDied","Data":"7eb6353a25e4f52de0dbf9a068364f808c70b26a4ff2e09500edea5839bd963f"} Oct 30 02:40:39.396689 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.396598 6275 scope.go:122] "RemoveContainer" containerID="e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c" Oct 30 02:40:39.415330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.415249 6275 scope.go:122] "RemoveContainer" containerID="3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e" Oct 30 02:40:39.426529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.426508 6275 scope.go:122] "RemoveContainer" containerID="e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c" Oct 30 02:40:39.426848 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:39.426827 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c\": container with ID starting with e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c not found: ID does not exist" containerID="e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c" Oct 30 02:40:39.426898 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.426856 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c"} err="failed to get container status \"e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c\": rpc error: code = NotFound desc = could not find container \"e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c\": container with ID starting with e8cf9f5e8081cbc02bad4f8765b54f6d222d9909717f51086fcc46d726f5333c not found: ID does not exist" Oct 30 02:40:39.426898 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.426874 6275 scope.go:122] "RemoveContainer" containerID="3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e" Oct 30 02:40:39.427133 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:39.427101 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e\": container with ID starting with 3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e not found: ID does not exist" containerID="3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e" Oct 30 02:40:39.427199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.427143 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e"} err="failed to get container status \"3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e\": rpc error: code = NotFound desc = could not find container \"3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e\": container with ID starting with 3ef74fbbffdbb187160a86f6377fb114303b127c5adfbe0d448056b8ee00da3e not found: ID does not exist" Oct 30 02:40:39.459933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.459902 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn"] Oct 30 02:40:39.470578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.470553 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4153e5-openshift-default-d76468489-mtrkn"] Oct 30 02:40:39.739728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:39.739651 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" path="/var/lib/kubelet/pods/68d1c41c-7055-41c5-9a20-3a1b41dd709b/volumes" Oct 30 02:40:40.198044 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:40.198004 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk"] Oct 30 02:40:40.228178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:40.228135 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8"] Oct 30 02:40:45.198662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:45.198615 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" containerName="payload-processing" containerID="cri-o://511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95" gracePeriod=25 Oct 30 02:40:45.229371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:45.229321 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerName="payload-pre-processing" containerID="cri-o://b4e80e8094d6176c60f578a61aad1904047b8d440c0f135fd641913720a013d2" gracePeriod=25 Oct 30 02:40:46.698101 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:40:46.698062 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:40:48.087529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:48.087499 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh"] Oct 30 02:40:48.124662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:48.124631 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs"] Oct 30 02:40:53.087775 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:53.087726 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerName="payload-processing" containerID="cri-o://8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1" gracePeriod=25 Oct 30 02:40:53.125626 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:53.125583 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerName="payload-pre-processing" containerID="cri-o://7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e" gracePeriod=25 Oct 30 02:40:53.292461 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:53.292421 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:53.292642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:53.292435 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:56.887391 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:56.887352 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:56.887777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:40:56.887382 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:03.291992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:03.291951 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:03.292524 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:03.291956 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:05.545383 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.545350 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:41:05.564379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.564316 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk\" (UniqueName: \"kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk\") pod \"b01d5808-adaf-4887-96f1-6319afa759b7\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " Oct 30 02:41:05.564581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.564504 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume\") pod \"b01d5808-adaf-4887-96f1-6319afa759b7\" (UID: \"b01d5808-adaf-4887-96f1-6319afa759b7\") " Oct 30 02:41:05.564885 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.564855 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume" pod "b01d5808-adaf-4887-96f1-6319afa759b7" (UID: "b01d5808-adaf-4887-96f1-6319afa759b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:05.565442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.565422 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d5808-adaf-4887-96f1-6319afa759b7-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:05.566655 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.566635 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk" pod "b01d5808-adaf-4887-96f1-6319afa759b7" (UID: "b01d5808-adaf-4887-96f1-6319afa759b7"). InnerVolumeSpecName "kube-api-access-fhbhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:05.578729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.578704 6275 generic.go:356] "Generic (PLEG): container finished" podID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerID="b4e80e8094d6176c60f578a61aad1904047b8d440c0f135fd641913720a013d2" exitCode=0 Oct 30 02:41:05.578816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.578788 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" event={"ID":"70b79155-0055-4bfa-9fc3-cdb24a0344d7","Type":"ContainerDied","Data":"b4e80e8094d6176c60f578a61aad1904047b8d440c0f135fd641913720a013d2"} Oct 30 02:41:05.580544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.580520 6275 generic.go:356] "Generic (PLEG): container finished" podID="b01d5808-adaf-4887-96f1-6319afa759b7" containerID="511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95" exitCode=0 Oct 30 02:41:05.580677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.580588 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" Oct 30 02:41:05.580813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.580586 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" event={"ID":"b01d5808-adaf-4887-96f1-6319afa759b7","Type":"ContainerDied","Data":"511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95"} Oct 30 02:41:05.580813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.580758 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk" event={"ID":"b01d5808-adaf-4887-96f1-6319afa759b7","Type":"ContainerDied","Data":"cc257c306cc3d38bee8cb7ab5e0fa7375f32f730b2e0ba9045ba442d1d96309e"} Oct 30 02:41:05.580813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.580786 6275 scope.go:122] "RemoveContainer" containerID="511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95" Oct 30 02:41:05.594277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.594187 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:41:05.594360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.594344 6275 scope.go:122] "RemoveContainer" containerID="511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95" Oct 30 02:41:05.594699 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:05.594676 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95\": container with ID starting with 511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95 not found: ID does not exist" containerID="511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95" Oct 30 02:41:05.594808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.594715 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95"} err="failed to get container status \"511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95\": rpc error: code = NotFound desc = could not find container \"511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95\": container with ID starting with 511c4c07b86cd9bcaec1eedace65404bb8ce65c681ff68cf1caf4e6a9aef6a95 not found: ID does not exist" Oct 30 02:41:05.630607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.630562 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk"] Oct 30 02:41:05.635643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.635606 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-17e747-6c68bf6668-ktzxk"] Oct 30 02:41:05.666722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.666692 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume\") pod \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " Oct 30 02:41:05.666904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.666799 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx\" (UniqueName: \"kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx\") pod \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\" (UID: \"70b79155-0055-4bfa-9fc3-cdb24a0344d7\") " Oct 30 02:41:05.667081 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.667059 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume" pod "70b79155-0055-4bfa-9fc3-cdb24a0344d7" (UID: "70b79155-0055-4bfa-9fc3-cdb24a0344d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:05.667399 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.667370 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhbhk\" (UniqueName: \"kubernetes.io/projected/b01d5808-adaf-4887-96f1-6319afa759b7-kube-api-access-fhbhk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:05.667399 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.667397 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b79155-0055-4bfa-9fc3-cdb24a0344d7-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:05.668970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.668952 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx" pod "70b79155-0055-4bfa-9fc3-cdb24a0344d7" (UID: "70b79155-0055-4bfa-9fc3-cdb24a0344d7"). InnerVolumeSpecName "kube-api-access-svzjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:05.740203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.740167 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" path="/var/lib/kubelet/pods/b01d5808-adaf-4887-96f1-6319afa759b7/volumes" Oct 30 02:41:05.769088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:05.769057 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svzjx\" (UniqueName: \"kubernetes.io/projected/70b79155-0055-4bfa-9fc3-cdb24a0344d7-kube-api-access-svzjx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:06.589664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.589634 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" Oct 30 02:41:06.590156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.589678 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8" event={"ID":"70b79155-0055-4bfa-9fc3-cdb24a0344d7","Type":"ContainerDied","Data":"59f9e2ec904ba442487c27b6c1851bf3b7188ef3375fd0972c273d3b1ea15382"} Oct 30 02:41:06.590156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.589747 6275 scope.go:122] "RemoveContainer" containerID="b4e80e8094d6176c60f578a61aad1904047b8d440c0f135fd641913720a013d2" Oct 30 02:41:06.637971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.637941 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8"] Oct 30 02:41:06.647946 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.647918 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-17e747-54ffc45d87-zpls8"] Oct 30 02:41:06.887154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.887029 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:06.887154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:06.887052 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:07.740819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:07.740786 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" path="/var/lib/kubelet/pods/70b79155-0055-4bfa-9fc3-cdb24a0344d7/volumes" Oct 30 02:41:08.345639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:08.345601 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts"] Oct 30 02:41:08.372239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:08.372198 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg"] Oct 30 02:41:13.346429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.346315 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" podUID="10fef1de-4941-4c89-91f4-dfb79be58160" containerName="payload-processing" containerID="cri-o://9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64" gracePeriod=25 Oct 30 02:41:13.374656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.372836 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" podUID="4dc2bec3-8673-438e-9e66-ae67425803aa" containerName="payload-pre-processing" containerID="cri-o://7a9d861a2299155301ac23bb630a670902c5f9d25c7c0f8c7f2398d888f62b18" gracePeriod=25 Oct 30 02:41:13.511114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.510777 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:41:13.515024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.514938 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:41:13.605938 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.605836 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume\") pod \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " Oct 30 02:41:13.606401 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.606355 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume\") pod \"f4fc8538-c30f-4242-9342-ed8661810e6a\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " Oct 30 02:41:13.606513 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.606402 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume" pod "3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" (UID: "3d3e07b5-6ab5-4e0a-ba02-4f50786777e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:13.606807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.606785 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume" pod "f4fc8538-c30f-4242-9342-ed8661810e6a" (UID: "f4fc8538-c30f-4242-9342-ed8661810e6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:13.606902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.606878 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl\" (UniqueName: \"kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl\") pod \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\" (UID: \"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0\") " Oct 30 02:41:13.606985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.606917 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95\" (UniqueName: \"kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95\") pod \"f4fc8538-c30f-4242-9342-ed8661810e6a\" (UID: \"f4fc8538-c30f-4242-9342-ed8661810e6a\") " Oct 30 02:41:13.608084 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.608040 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.608234 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.608100 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fc8538-c30f-4242-9342-ed8661810e6a-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.610561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.610533 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95" pod "f4fc8538-c30f-4242-9342-ed8661810e6a" (UID: "f4fc8538-c30f-4242-9342-ed8661810e6a"). InnerVolumeSpecName "kube-api-access-khr95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:13.611809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.611768 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl" pod "3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" (UID: "3d3e07b5-6ab5-4e0a-ba02-4f50786777e0"). InnerVolumeSpecName "kube-api-access-pzzfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:13.637556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.637175 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:41:13.645093 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.645058 6275 generic.go:356] "Generic (PLEG): container finished" podID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerID="7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e" exitCode=0 Oct 30 02:41:13.645281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.645104 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" Oct 30 02:41:13.645281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.645208 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" event={"ID":"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0","Type":"ContainerDied","Data":"7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e"} Oct 30 02:41:13.645281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.645257 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs" event={"ID":"3d3e07b5-6ab5-4e0a-ba02-4f50786777e0","Type":"ContainerDied","Data":"446e10db009b58e0517e8ac4cda761af510ff31d0322c3799093b7725e6e3c9d"} Oct 30 02:41:13.645439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.645287 6275 scope.go:122] "RemoveContainer" containerID="7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e" Oct 30 02:41:13.647419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.647395 6275 generic.go:356] "Generic (PLEG): container finished" podID="10fef1de-4941-4c89-91f4-dfb79be58160" containerID="9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64" exitCode=0 Oct 30 02:41:13.647520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.647433 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" Oct 30 02:41:13.647595 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.647540 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" event={"ID":"10fef1de-4941-4c89-91f4-dfb79be58160","Type":"ContainerDied","Data":"9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64"} Oct 30 02:41:13.647833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.647804 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts" event={"ID":"10fef1de-4941-4c89-91f4-dfb79be58160","Type":"ContainerDied","Data":"7a49c299cf7c1fe9ce2753fad2db66456bbf69e69d8f06c053f5126674244335"} Oct 30 02:41:13.650610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.650555 6275 generic.go:356] "Generic (PLEG): container finished" podID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerID="8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1" exitCode=0 Oct 30 02:41:13.650610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.650605 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" event={"ID":"f4fc8538-c30f-4242-9342-ed8661810e6a","Type":"ContainerDied","Data":"8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1"} Oct 30 02:41:13.651187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.650643 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" event={"ID":"f4fc8538-c30f-4242-9342-ed8661810e6a","Type":"ContainerDied","Data":"db5ffda18fd6aa712b8d63c6b810a4c6a8a785090858e12580bcb49ad49f5dea"} Oct 30 02:41:13.651187 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.650591 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh" Oct 30 02:41:13.658222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.657734 6275 generic.go:356] "Generic (PLEG): container finished" podID="4dc2bec3-8673-438e-9e66-ae67425803aa" containerID="7a9d861a2299155301ac23bb630a670902c5f9d25c7c0f8c7f2398d888f62b18" exitCode=0 Oct 30 02:41:13.658222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.657787 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" event={"ID":"4dc2bec3-8673-438e-9e66-ae67425803aa","Type":"ContainerDied","Data":"7a9d861a2299155301ac23bb630a670902c5f9d25c7c0f8c7f2398d888f62b18"} Oct 30 02:41:13.662677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.662596 6275 scope.go:122] "RemoveContainer" containerID="7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e" Oct 30 02:41:13.663430 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:13.663406 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e\": container with ID starting with 7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e not found: ID does not exist" containerID="7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e" Oct 30 02:41:13.663538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.663440 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e"} err="failed to get container status \"7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e\": rpc error: code = NotFound desc = could not find container \"7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e\": container with ID starting with 7c76694e7dfc288a0a06f2b950bf7a4ce72ea83445fc8966906b50bc9ac8403e not found: ID does not exist" Oct 30 02:41:13.663538 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.663462 6275 scope.go:122] "RemoveContainer" containerID="9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64" Oct 30 02:41:13.677081 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.677056 6275 scope.go:122] "RemoveContainer" containerID="9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64" Oct 30 02:41:13.677937 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:13.677908 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64\": container with ID starting with 9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64 not found: ID does not exist" containerID="9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64" Oct 30 02:41:13.678041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.677946 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64"} err="failed to get container status \"9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64\": rpc error: code = NotFound desc = could not find container \"9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64\": container with ID starting with 9b1e7c195e8142556166dda2cdc6f32ef543b91af67c4a557f32e19b033cac64 not found: ID does not exist" Oct 30 02:41:13.678041 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.677964 6275 scope.go:122] "RemoveContainer" containerID="8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1" Oct 30 02:41:13.691265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.691242 6275 scope.go:122] "RemoveContainer" containerID="8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1" Oct 30 02:41:13.692270 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:13.692233 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1\": container with ID starting with 8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1 not found: ID does not exist" containerID="8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1" Oct 30 02:41:13.692388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.692267 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1"} err="failed to get container status \"8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1\": rpc error: code = NotFound desc = could not find container \"8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1\": container with ID starting with 8af063b7f87904ad03b22e30f49d4fe2d5bcf2bbbe2608f1054ea51acffc28a1 not found: ID does not exist" Oct 30 02:41:13.698520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.698493 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:41:13.709296 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.709263 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume\") pod \"10fef1de-4941-4c89-91f4-dfb79be58160\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " Oct 30 02:41:13.709413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.709310 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v\" (UniqueName: \"kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v\") pod \"10fef1de-4941-4c89-91f4-dfb79be58160\" (UID: \"10fef1de-4941-4c89-91f4-dfb79be58160\") " Oct 30 02:41:13.709707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.709673 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume" pod "10fef1de-4941-4c89-91f4-dfb79be58160" (UID: "10fef1de-4941-4c89-91f4-dfb79be58160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:13.710015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.709992 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fef1de-4941-4c89-91f4-dfb79be58160-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.710114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.710020 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzzfl\" (UniqueName: \"kubernetes.io/projected/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0-kube-api-access-pzzfl\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.710114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.710035 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khr95\" (UniqueName: \"kubernetes.io/projected/f4fc8538-c30f-4242-9342-ed8661810e6a-kube-api-access-khr95\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.711516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.711492 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v" pod "10fef1de-4941-4c89-91f4-dfb79be58160" (UID: "10fef1de-4941-4c89-91f4-dfb79be58160"). InnerVolumeSpecName "kube-api-access-7rb8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:13.722222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.722195 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs"] Oct 30 02:41:13.746006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.745981 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0ce950-84cb98fcf5-gfpbs"] Oct 30 02:41:13.782595 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.782567 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh"] Oct 30 02:41:13.792159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.792096 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0ce950-5db457fb88-q2nzh"] Oct 30 02:41:13.810845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.810814 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume\") pod \"4dc2bec3-8673-438e-9e66-ae67425803aa\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " Oct 30 02:41:13.811011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.810997 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w\" (UniqueName: \"kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w\") pod \"4dc2bec3-8673-438e-9e66-ae67425803aa\" (UID: \"4dc2bec3-8673-438e-9e66-ae67425803aa\") " Oct 30 02:41:13.811287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.811254 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume" pod "4dc2bec3-8673-438e-9e66-ae67425803aa" (UID: "4dc2bec3-8673-438e-9e66-ae67425803aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:13.811591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.811573 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc2bec3-8673-438e-9e66-ae67425803aa-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.811657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.811597 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rb8v\" (UniqueName: \"kubernetes.io/projected/10fef1de-4941-4c89-91f4-dfb79be58160-kube-api-access-7rb8v\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.813722 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.813700 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w" pod "4dc2bec3-8673-438e-9e66-ae67425803aa" (UID: "4dc2bec3-8673-438e-9e66-ae67425803aa"). InnerVolumeSpecName "kube-api-access-qq74w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:13.912835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.912740 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq74w\" (UniqueName: \"kubernetes.io/projected/4dc2bec3-8673-438e-9e66-ae67425803aa-kube-api-access-qq74w\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:13.987043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.987000 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts"] Oct 30 02:41:13.997541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:13.997506 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-edae4317-69d497b588-dhfts"] Oct 30 02:41:14.670205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:14.670173 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" Oct 30 02:41:14.670205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:14.670187 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg" event={"ID":"4dc2bec3-8673-438e-9e66-ae67425803aa","Type":"ContainerDied","Data":"42727ba8201097e0455c541cc67c7ee8e78721abb7b4855850da58053135f2eb"} Oct 30 02:41:14.670764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:14.670254 6275 scope.go:122] "RemoveContainer" containerID="7a9d861a2299155301ac23bb630a670902c5f9d25c7c0f8c7f2398d888f62b18" Oct 30 02:41:14.724789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:14.724753 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg"] Oct 30 02:41:14.734803 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:14.734758 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-edae4317-8477bbf575-tjdfg"] Oct 30 02:41:15.746330 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:15.746297 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fef1de-4941-4c89-91f4-dfb79be58160" path="/var/lib/kubelet/pods/10fef1de-4941-4c89-91f4-dfb79be58160/volumes" Oct 30 02:41:15.746748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:15.746727 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" path="/var/lib/kubelet/pods/3d3e07b5-6ab5-4e0a-ba02-4f50786777e0/volumes" Oct 30 02:41:15.747447 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:15.747425 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc2bec3-8673-438e-9e66-ae67425803aa" path="/var/lib/kubelet/pods/4dc2bec3-8673-438e-9e66-ae67425803aa/volumes" Oct 30 02:41:15.747898 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:15.747885 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" path="/var/lib/kubelet/pods/f4fc8538-c30f-4242-9342-ed8661810e6a/volumes" Oct 30 02:41:16.698730 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:16.698692 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:41:18.684535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.684504 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx"] Oct 30 02:41:18.685182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685167 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerName="payload-pre-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685183 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerName="payload-pre-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685197 6275 cpu_manager.go:414] "containerMap: removing container" podUID="10fef1de-4941-4c89-91f4-dfb79be58160" containerName="payload-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685203 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fef1de-4941-4c89-91f4-dfb79be58160" containerName="payload-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685214 6275 cpu_manager.go:414] "containerMap: removing container" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerName="payload-pre-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685219 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerName="payload-pre-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685226 6275 cpu_manager.go:414] "containerMap: removing container" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerName="payload-pre-processing" Oct 30 02:41:18.685229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685231 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685242 6275 cpu_manager.go:414] "containerMap: removing container" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685248 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685255 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685259 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685272 6275 cpu_manager.go:414] "containerMap: removing container" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685276 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685283 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4dc2bec3-8673-438e-9e66-ae67425803aa" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685288 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc2bec3-8673-438e-9e66-ae67425803aa" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685297 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1155a223-6a0e-41f5-8966-0365df3ad93a" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685300 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1155a223-6a0e-41f5-8966-0365df3ad93a" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685306 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ea512869-be10-4520-a3e1-6d1827de552f" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685311 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea512869-be10-4520-a3e1-6d1827de552f" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685323 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ae9b6424-2631-48fb-a12c-42dd5303a752" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685327 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9b6424-2631-48fb-a12c-42dd5303a752" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685335 6275 cpu_manager.go:414] "containerMap: removing container" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685339 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685348 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685352 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685359 6275 cpu_manager.go:414] "containerMap: removing container" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685364 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685371 6275 cpu_manager.go:414] "containerMap: removing container" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685375 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" containerName="payload-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685387 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685391 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685396 6275 cpu_manager.go:414] "containerMap: removing container" podUID="428a6289-6cc8-4c54-8103-9833110bb7c3" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685401 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428a6289-6cc8-4c54-8103-9833110bb7c3" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685406 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685411 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerName="payload-pre-processing" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685422 6275 cpu_manager.go:414] "containerMap: removing container" podUID="600da90f-44d7-45ec-8dab-41133df0f8cc" containerName="maas-api" Oct 30 02:41:18.685464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685426 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600da90f-44d7-45ec-8dab-41133df0f8cc" containerName="maas-api" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685521 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9b6424-2631-48fb-a12c-42dd5303a752" containerName="maas-api" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685528 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5421aa-5367-446c-b3f2-8261bf43d585" containerName="istio-proxy" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685535 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52d08b9-a76e-4c95-a64d-3b2dbe73428d" containerName="payload-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685540 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685546 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01d5808-adaf-4887-96f1-6319afa759b7" containerName="payload-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685555 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b82dfb5-2853-4663-8c19-c83c5d32ec0b" containerName="payload-pre-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685561 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="428a6289-6cc8-4c54-8103-9833110bb7c3" containerName="maas-api" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685567 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8515cfeb-db30-420b-95f3-7f84e37f1e3a" containerName="payload-pre-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685572 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="600da90f-44d7-45ec-8dab-41133df0f8cc" containerName="maas-api" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685581 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b095cab-6798-4fcc-960d-fdf54d7e6522" containerName="payload-pre-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685588 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fef1de-4941-4c89-91f4-dfb79be58160" containerName="payload-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685594 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b79155-0055-4bfa-9fc3-cdb24a0344d7" containerName="payload-pre-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685600 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3e07b5-6ab5-4e0a-ba02-4f50786777e0" containerName="payload-pre-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685607 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fc8538-c30f-4242-9342-ed8661810e6a" containerName="payload-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685612 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eae7013-7f1b-4066-babf-ed2ee657212a" containerName="payload-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685620 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc2bec3-8673-438e-9e66-ae67425803aa" containerName="payload-pre-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685626 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1155a223-6a0e-41f5-8966-0365df3ad93a" containerName="payload-processing" Oct 30 02:41:18.686414 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.685632 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea512869-be10-4520-a3e1-6d1827de552f" containerName="maas-api" Oct 30 02:41:18.690764 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.690743 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.720859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.720823 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"kube-root-ca.crt\"" Oct 30 02:41:18.720859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.720823 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"openshift-service-ca.crt\"" Oct 30 02:41:18.721051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.720879 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"rate-test-model-24fb97-kserve-self-signed-certs\"" Oct 30 02:41:18.721220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.721202 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"default-dockercfg-5gtr5\"" Oct 30 02:41:18.724821 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.724796 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx"] Oct 30 02:41:18.786625 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.786589 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.786818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.786635 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dzq\" (UniqueName: \"kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.786818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.786659 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.786818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.786800 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.786970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.786899 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.786970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.786962 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.888757 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.888714 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.888978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.888819 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.888978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.888886 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.888978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.888939 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.888978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.888969 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dzq\" (UniqueName: \"kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.889278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.888990 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.889388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.889366 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.889434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.889383 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.889476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.889449 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.891343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.891314 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.891614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.891588 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:18.911224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:18.911189 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dzq\" (UniqueName: \"kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq\") pod \"rate-test-model-24fb97-kserve-67dcd9989-xpvtx\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:19.008224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:19.008087 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:19.342671 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:19.342642 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx"] Oct 30 02:41:19.345535 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:41:19.345492 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1f64997_cd28_4d25_ad03_ddd332ac48fb.slice/crio-927eb73731cab94ba1c798bff8d31eed2d7c2dea60e8062f12ce90cd506904a4 WatchSource:0}: Error finding container 927eb73731cab94ba1c798bff8d31eed2d7c2dea60e8062f12ce90cd506904a4: Status 404 returned error can't find the container with id 927eb73731cab94ba1c798bff8d31eed2d7c2dea60e8062f12ce90cd506904a4 Oct 30 02:41:19.708665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:19.708573 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" event={"ID":"a1f64997-cd28-4d25-ad03-ddd332ac48fb","Type":"ContainerStarted","Data":"6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3"} Oct 30 02:41:19.708665 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:19.708617 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" event={"ID":"a1f64997-cd28-4d25-ad03-ddd332ac48fb","Type":"ContainerStarted","Data":"927eb73731cab94ba1c798bff8d31eed2d7c2dea60e8062f12ce90cd506904a4"} Oct 30 02:41:19.751014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:19.750953 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" podStartSLOduration=1.750934266 podStartE2EDuration="1.750934266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:19.747632388 +0000 UTC m=+4308.139440022" watchObservedRunningTime="2026-10-30 02:41:19.750934266 +0000 UTC m=+4308.142741900" Oct 30 02:41:20.009045 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:20.009014 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:25.115072 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.115036 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv"] Oct 30 02:41:25.115550 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.115257 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" podUID="11837e56-a8e4-46f7-b265-09a108e54fa9" containerName="maas-api" containerID="cri-o://8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f" gracePeriod=30 Oct 30 02:41:25.440348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.440319 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:41:25.509149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.508581 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs\") pod \"11837e56-a8e4-46f7-b265-09a108e54fa9\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " Oct 30 02:41:25.509149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.508652 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca\") pod \"11837e56-a8e4-46f7-b265-09a108e54fa9\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " Oct 30 02:41:25.509149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.508707 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\") pod \"11837e56-a8e4-46f7-b265-09a108e54fa9\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " Oct 30 02:41:25.509149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.508900 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b\" (UniqueName: \"kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b\") pod \"11837e56-a8e4-46f7-b265-09a108e54fa9\" (UID: \"11837e56-a8e4-46f7-b265-09a108e54fa9\") " Oct 30 02:41:25.522749 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.514202 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca" pod "11837e56-a8e4-46f7-b265-09a108e54fa9" (UID: "11837e56-a8e4-46f7-b265-09a108e54fa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:25.594964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.592708 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs" pod "11837e56-a8e4-46f7-b265-09a108e54fa9" (UID: "11837e56-a8e4-46f7-b265-09a108e54fa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:25.594964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.592909 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b" pod "11837e56-a8e4-46f7-b265-09a108e54fa9" (UID: "11837e56-a8e4-46f7-b265-09a108e54fa9"). InnerVolumeSpecName "kube-api-access-v468b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:25.618320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.600738 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls" pod "11837e56-a8e4-46f7-b265-09a108e54fa9" (UID: "11837e56-a8e4-46f7-b265-09a108e54fa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:25.618320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.617183 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v468b\" (UniqueName: \"kubernetes.io/projected/11837e56-a8e4-46f7-b265-09a108e54fa9-kube-api-access-v468b\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:25.618320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.617218 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:25.618320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.617233 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11837e56-a8e4-46f7-b265-09a108e54fa9-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:25.618320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.617245 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11837e56-a8e4-46f7-b265-09a108e54fa9-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:41:25.752426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.752389 6275 generic.go:356] "Generic (PLEG): container finished" podID="11837e56-a8e4-46f7-b265-09a108e54fa9" containerID="8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f" exitCode=0 Oct 30 02:41:25.752567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.752431 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" Oct 30 02:41:25.752567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.752467 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" event={"ID":"11837e56-a8e4-46f7-b265-09a108e54fa9","Type":"ContainerDied","Data":"8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f"} Oct 30 02:41:25.752567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.752512 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv" event={"ID":"11837e56-a8e4-46f7-b265-09a108e54fa9","Type":"ContainerDied","Data":"06b3a1bab32ece64fa75f1fe4c2b67d5872fb509a1c088767d628a3b64ca767e"} Oct 30 02:41:25.752567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.752532 6275 scope.go:122] "RemoveContainer" containerID="8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f" Oct 30 02:41:25.764484 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.764466 6275 scope.go:122] "RemoveContainer" containerID="8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f" Oct 30 02:41:25.764785 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:25.764761 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f\": container with ID starting with 8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f not found: ID does not exist" containerID="8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f" Oct 30 02:41:25.764848 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.764792 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f"} err="failed to get container status \"8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f\": rpc error: code = NotFound desc = could not find container \"8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f\": container with ID starting with 8fa3e2651fa2612f905d21aa4286b3ef3509bfe2bb50ecc78622a11bef53470f not found: ID does not exist" Oct 30 02:41:25.823787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.823753 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv"] Oct 30 02:41:25.840102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:25.840071 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-edae4317-6744969b7c-gkpcv"] Oct 30 02:41:27.758154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:27.758099 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11837e56-a8e4-46f7-b265-09a108e54fa9" path="/var/lib/kubelet/pods/11837e56-a8e4-46f7-b265-09a108e54fa9/volumes" Oct 30 02:41:29.008492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:29.008454 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:29.021289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:29.021265 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:29.794910 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:29.794876 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:41:41.620741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.620704 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p"] Oct 30 02:41:41.621631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.621606 6275 cpu_manager.go:414] "containerMap: removing container" podUID="11837e56-a8e4-46f7-b265-09a108e54fa9" containerName="maas-api" Oct 30 02:41:41.621631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.621631 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11837e56-a8e4-46f7-b265-09a108e54fa9" containerName="maas-api" Oct 30 02:41:41.621823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.621778 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="11837e56-a8e4-46f7-b265-09a108e54fa9" containerName="maas-api" Oct 30 02:41:41.621823 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.621794 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d1c41c-7055-41c5-9a20-3a1b41dd709b" containerName="istio-proxy" Oct 30 02:41:41.625529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.625509 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.650160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.648948 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"openshift-service-ca.crt\"" Oct 30 02:41:41.650160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.649391 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"kube-root-ca.crt\"" Oct 30 02:41:41.650160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.649500 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"default-dockercfg-drkbz\"" Oct 30 02:41:41.650160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.649580 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"rate-test-model-584ec5-kserve-self-signed-certs\"" Oct 30 02:41:41.654965 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.654935 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p"] Oct 30 02:41:41.742861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.742827 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.742861 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.742863 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.743089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.742988 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.743089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.743038 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.743180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.743141 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkbx\" (UniqueName: \"kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.743180 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.743169 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844331 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844372 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844494 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844634 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844553 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844629 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkbx\" (UniqueName: \"kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844662 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844838 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844872 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.844928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.844917 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.846954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.846929 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.847276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.847255 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.881509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.881393 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkbx\" (UniqueName: \"kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx\") pod \"rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:41.939504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:41.939468 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:42.257464 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:42.256105 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p"] Oct 30 02:41:42.260468 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:41:42.260438 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b5a4a6d_5e8e_480e_a35d_1c339fb4a69a.slice/crio-6c6f033f3ea411a6c7143776812cc0db956059e7d67723e6d80160720b82e355 WatchSource:0}: Error finding container 6c6f033f3ea411a6c7143776812cc0db956059e7d67723e6d80160720b82e355: Status 404 returned error can't find the container with id 6c6f033f3ea411a6c7143776812cc0db956059e7d67723e6d80160720b82e355 Oct 30 02:41:42.888248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:42.888212 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" event={"ID":"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a","Type":"ContainerStarted","Data":"75caa579ea80ae1a36e5710af739089d0430fff3d655cbd0a8a68f00f2c081ba"} Oct 30 02:41:42.888641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:42.888258 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" event={"ID":"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a","Type":"ContainerStarted","Data":"6c6f033f3ea411a6c7143776812cc0db956059e7d67723e6d80160720b82e355"} Oct 30 02:41:42.928804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:42.928743 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" podStartSLOduration=1.9287306389999999 podStartE2EDuration="1.928730639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:42.926253913 +0000 UTC m=+4331.318061546" watchObservedRunningTime="2026-10-30 02:41:42.928730639 +0000 UTC m=+4331.320538271" Oct 30 02:41:42.940429 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:42.940399 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:45.490396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:45.490355 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9"] Oct 30 02:41:45.528752 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:45.528721 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8"] Oct 30 02:41:46.698564 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:41:46.698518 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:41:50.491614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:50.491570 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerName="payload-processing" containerID="cri-o://f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473" gracePeriod=25 Oct 30 02:41:50.529668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:50.529628 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" containerName="payload-pre-processing" containerID="cri-o://a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333" gracePeriod=25 Oct 30 02:41:51.940701 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:51.940660 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:51.954149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:51.954076 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:51.965318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:51.965288 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:41:58.898372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:58.898333 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:58.898827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:41:58.898331 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:42:01.261146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:01.261081 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k"] Oct 30 02:42:01.292908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:01.292867 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9"] Oct 30 02:42:06.262135 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.262087 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" podUID="90615baf-1842-45de-b5ed-57f5a59d02f4" containerName="payload-processing" containerID="cri-o://cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59" gracePeriod=25 Oct 30 02:42:06.294275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.294226 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" podUID="05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" containerName="payload-pre-processing" containerID="cri-o://f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832" gracePeriod=25 Oct 30 02:42:06.563532 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.563501 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:42:06.610147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.610091 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:42:06.648920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.648858 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume\") pod \"90615baf-1842-45de-b5ed-57f5a59d02f4\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " Oct 30 02:42:06.649398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.649368 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume" pod "90615baf-1842-45de-b5ed-57f5a59d02f4" (UID: "90615baf-1842-45de-b5ed-57f5a59d02f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:06.650611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.649960 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw\" (UniqueName: \"kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw\") pod \"90615baf-1842-45de-b5ed-57f5a59d02f4\" (UID: \"90615baf-1842-45de-b5ed-57f5a59d02f4\") " Oct 30 02:42:06.650611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.650586 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90615baf-1842-45de-b5ed-57f5a59d02f4-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:06.652578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.652554 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw" pod "90615baf-1842-45de-b5ed-57f5a59d02f4" (UID: "90615baf-1842-45de-b5ed-57f5a59d02f4"). InnerVolumeSpecName "kube-api-access-dhvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:06.751462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.751426 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr\" (UniqueName: \"kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr\") pod \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " Oct 30 02:42:06.751750 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.751662 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume\") pod \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\" (UID: \"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e\") " Oct 30 02:42:06.752037 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.752003 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume" pod "05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" (UID: "05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:06.752223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.752208 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:06.752223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.752221 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvtw\" (UniqueName: \"kubernetes.io/projected/90615baf-1842-45de-b5ed-57f5a59d02f4-kube-api-access-dhvtw\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:06.753726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.753698 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr" pod "05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" (UID: "05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e"). InnerVolumeSpecName "kube-api-access-62wqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:06.854036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:06.853947 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62wqr\" (UniqueName: \"kubernetes.io/projected/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e-kube-api-access-62wqr\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:07.051396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.051360 6275 generic.go:356] "Generic (PLEG): container finished" podID="05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" containerID="f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832" exitCode=0 Oct 30 02:42:07.051591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.051407 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" Oct 30 02:42:07.051591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.051438 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" event={"ID":"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e","Type":"ContainerDied","Data":"f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832"} Oct 30 02:42:07.051591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.051499 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9" event={"ID":"05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e","Type":"ContainerDied","Data":"18a8cf36528f27b5ab57d82ba33615221155b9e27c2fed1b39f955656d57ee35"} Oct 30 02:42:07.051591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.051522 6275 scope.go:122] "RemoveContainer" containerID="f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832" Oct 30 02:42:07.052952 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.052927 6275 generic.go:356] "Generic (PLEG): container finished" podID="90615baf-1842-45de-b5ed-57f5a59d02f4" containerID="cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59" exitCode=0 Oct 30 02:42:07.053095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.052971 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" Oct 30 02:42:07.053095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.053007 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" event={"ID":"90615baf-1842-45de-b5ed-57f5a59d02f4","Type":"ContainerDied","Data":"cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59"} Oct 30 02:42:07.053095 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.053043 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k" event={"ID":"90615baf-1842-45de-b5ed-57f5a59d02f4","Type":"ContainerDied","Data":"e9be5f46d9a7250c2787df3422e9855a84747f4405ec829c014e1b8c8066bf0b"} Oct 30 02:42:07.063453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.063432 6275 scope.go:122] "RemoveContainer" containerID="f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832" Oct 30 02:42:07.063769 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:07.063743 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832\": container with ID starting with f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832 not found: ID does not exist" containerID="f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832" Oct 30 02:42:07.063881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.063776 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832"} err="failed to get container status \"f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832\": rpc error: code = NotFound desc = could not find container \"f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832\": container with ID starting with f4fff97884fba9eaa8346f66a08f65e5740f769be1f93399d3509934b6458832 not found: ID does not exist" Oct 30 02:42:07.063881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.063800 6275 scope.go:122] "RemoveContainer" containerID="cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59" Oct 30 02:42:07.073399 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.073383 6275 scope.go:122] "RemoveContainer" containerID="cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59" Oct 30 02:42:07.073714 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:07.073692 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59\": container with ID starting with cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59 not found: ID does not exist" containerID="cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59" Oct 30 02:42:07.073828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.073717 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59"} err="failed to get container status \"cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59\": rpc error: code = NotFound desc = could not find container \"cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59\": container with ID starting with cfcb99846117624fe84dee451a4cda3ef014d60f9fc7c755dce26e8e4af7cb59 not found: ID does not exist" Oct 30 02:42:07.098554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.098530 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9"] Oct 30 02:42:07.109347 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.109291 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a00b8-7449c77959-cg5r9"] Oct 30 02:42:07.153019 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.152983 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k"] Oct 30 02:42:07.156986 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.156957 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a00b8-7f465f49db-8tg6k"] Oct 30 02:42:07.741958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.741925 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" path="/var/lib/kubelet/pods/05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e/volumes" Oct 30 02:42:07.742440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:07.742425 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90615baf-1842-45de-b5ed-57f5a59d02f4" path="/var/lib/kubelet/pods/90615baf-1842-45de-b5ed-57f5a59d02f4/volumes" Oct 30 02:42:08.898983 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:08.898946 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:42:08.899440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:08.898943 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:42:09.584488 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:09.584453 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7"] Oct 30 02:42:09.584725 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:09.584695 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" podUID="69742b2d-68ad-4041-af3c-c80740eca721" containerName="maas-api" containerID="cri-o://68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23" gracePeriod=30 Oct 30 02:42:09.911009 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:09.910975 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 02:42:10.093320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.093272 6275 generic.go:356] "Generic (PLEG): container finished" podID="69742b2d-68ad-4041-af3c-c80740eca721" containerID="68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23" exitCode=0 Oct 30 02:42:10.093320 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.093316 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" Oct 30 02:42:10.093593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.093352 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" event={"ID":"69742b2d-68ad-4041-af3c-c80740eca721","Type":"ContainerDied","Data":"68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23"} Oct 30 02:42:10.093593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.093406 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7" event={"ID":"69742b2d-68ad-4041-af3c-c80740eca721","Type":"ContainerDied","Data":"d39acb5b98e2f555414baa050c7d0387a24258a9b01e280e317336594d145611"} Oct 30 02:42:10.093593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.093435 6275 scope.go:122] "RemoveContainer" containerID="68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23" Oct 30 02:42:10.097534 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.097509 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca\") pod \"69742b2d-68ad-4041-af3c-c80740eca721\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " Oct 30 02:42:10.097695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.097578 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls\") pod \"69742b2d-68ad-4041-af3c-c80740eca721\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " Oct 30 02:42:10.097695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.097628 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs\") pod \"69742b2d-68ad-4041-af3c-c80740eca721\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " Oct 30 02:42:10.097695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.097652 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7\" (UniqueName: \"kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7\") pod \"69742b2d-68ad-4041-af3c-c80740eca721\" (UID: \"69742b2d-68ad-4041-af3c-c80740eca721\") " Oct 30 02:42:10.097917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.097892 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca" pod "69742b2d-68ad-4041-af3c-c80740eca721" (UID: "69742b2d-68ad-4041-af3c-c80740eca721"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:10.098346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.098321 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69742b2d-68ad-4041-af3c-c80740eca721-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:10.100116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.100090 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7" pod "69742b2d-68ad-4041-af3c-c80740eca721" (UID: "69742b2d-68ad-4041-af3c-c80740eca721"). InnerVolumeSpecName "kube-api-access-5gzj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:10.101961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.101938 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs" pod "69742b2d-68ad-4041-af3c-c80740eca721" (UID: "69742b2d-68ad-4041-af3c-c80740eca721"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:10.102287 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.102258 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls" pod "69742b2d-68ad-4041-af3c-c80740eca721" (UID: "69742b2d-68ad-4041-af3c-c80740eca721"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:10.105224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.105199 6275 scope.go:122] "RemoveContainer" containerID="68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23" Oct 30 02:42:10.105587 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:10.105566 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23\": container with ID starting with 68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23 not found: ID does not exist" containerID="68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23" Oct 30 02:42:10.105653 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.105597 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23"} err="failed to get container status \"68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23\": rpc error: code = NotFound desc = could not find container \"68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23\": container with ID starting with 68f54a1e5b20967f926a54b553240e98bb09ee6b425636d757db66574c804f23 not found: ID does not exist" Oct 30 02:42:10.199684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.199603 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:10.199684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.199631 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69742b2d-68ad-4041-af3c-c80740eca721-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:10.199684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.199642 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzj7\" (UniqueName: \"kubernetes.io/projected/69742b2d-68ad-4041-af3c-c80740eca721-kube-api-access-5gzj7\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:10.460509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.460410 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7"] Oct 30 02:42:10.464572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.464541 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b93d51-67fcdf7dc8-5rrd7"] Oct 30 02:42:10.831422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.831396 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 02:42:10.864894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:10.864867 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 02:42:11.010889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.010861 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd\" (UniqueName: \"kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd\") pod \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " Oct 30 02:42:11.011353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.010962 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume\") pod \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\" (UID: \"fbd25775-c6d6-4dcd-b19e-5db7800b899c\") " Oct 30 02:42:11.011353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.011046 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq\" (UniqueName: \"kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq\") pod \"8cdf999e-4537-4613-896b-7c9736e8136e\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " Oct 30 02:42:11.011353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.011157 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume\") pod \"8cdf999e-4537-4613-896b-7c9736e8136e\" (UID: \"8cdf999e-4537-4613-896b-7c9736e8136e\") " Oct 30 02:42:11.011557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.011443 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume" pod "fbd25775-c6d6-4dcd-b19e-5db7800b899c" (UID: "fbd25775-c6d6-4dcd-b19e-5db7800b899c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:11.011620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.011590 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume" pod "8cdf999e-4537-4613-896b-7c9736e8136e" (UID: "8cdf999e-4537-4613-896b-7c9736e8136e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:11.011892 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.011873 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdf999e-4537-4613-896b-7c9736e8136e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:11.011966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.011894 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd25775-c6d6-4dcd-b19e-5db7800b899c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:11.013374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.013346 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd" pod "fbd25775-c6d6-4dcd-b19e-5db7800b899c" (UID: "fbd25775-c6d6-4dcd-b19e-5db7800b899c"). InnerVolumeSpecName "kube-api-access-6cfvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:11.013723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.013700 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq" pod "8cdf999e-4537-4613-896b-7c9736e8136e" (UID: "8cdf999e-4537-4613-896b-7c9736e8136e"). InnerVolumeSpecName "kube-api-access-4jhgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:11.102542 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.102504 6275 generic.go:356] "Generic (PLEG): container finished" podID="8cdf999e-4537-4613-896b-7c9736e8136e" containerID="a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333" exitCode=0 Oct 30 02:42:11.102733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.102557 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" Oct 30 02:42:11.102733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.102601 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" event={"ID":"8cdf999e-4537-4613-896b-7c9736e8136e","Type":"ContainerDied","Data":"a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333"} Oct 30 02:42:11.102909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.102879 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8" event={"ID":"8cdf999e-4537-4613-896b-7c9736e8136e","Type":"ContainerDied","Data":"d76afe4f9722d65f7cb2f779775f2d815d13c062c81e22093a86393a92749ab3"} Oct 30 02:42:11.103062 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.103042 6275 scope.go:122] "RemoveContainer" containerID="a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333" Oct 30 02:42:11.113962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.107613 6275 generic.go:356] "Generic (PLEG): container finished" podID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerID="f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473" exitCode=0 Oct 30 02:42:11.113962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.107677 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" event={"ID":"fbd25775-c6d6-4dcd-b19e-5db7800b899c","Type":"ContainerDied","Data":"f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473"} Oct 30 02:42:11.113962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.107717 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" event={"ID":"fbd25775-c6d6-4dcd-b19e-5db7800b899c","Type":"ContainerDied","Data":"0c7e8db7b806acfbd83d312e881df9ce17123540e5ccb97f3d3dfc986710fe17"} Oct 30 02:42:11.113962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.107789 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9" Oct 30 02:42:11.113962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.112832 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jhgq\" (UniqueName: \"kubernetes.io/projected/8cdf999e-4537-4613-896b-7c9736e8136e-kube-api-access-4jhgq\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:11.113962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.112855 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfvd\" (UniqueName: \"kubernetes.io/projected/fbd25775-c6d6-4dcd-b19e-5db7800b899c-kube-api-access-6cfvd\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:11.122305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.122282 6275 scope.go:122] "RemoveContainer" containerID="a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333" Oct 30 02:42:11.122714 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:11.122686 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333\": container with ID starting with a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333 not found: ID does not exist" containerID="a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333" Oct 30 02:42:11.122817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.122724 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333"} err="failed to get container status \"a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333\": rpc error: code = NotFound desc = could not find container \"a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333\": container with ID starting with a13b4f0520ba1ed8f103ff23c2a9db7801ced7188c6fb4722042ecb510894333 not found: ID does not exist" Oct 30 02:42:11.122817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.122743 6275 scope.go:122] "RemoveContainer" containerID="f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473" Oct 30 02:42:11.133870 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.133849 6275 scope.go:122] "RemoveContainer" containerID="f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473" Oct 30 02:42:11.134254 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:11.134221 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473\": container with ID starting with f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473 not found: ID does not exist" containerID="f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473" Oct 30 02:42:11.134375 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.134266 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473"} err="failed to get container status \"f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473\": rpc error: code = NotFound desc = could not find container \"f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473\": container with ID starting with f6393b7834e8cd4e7df259290974ad4238f697a2d604c3f4fe975caa9abe6473 not found: ID does not exist" Oct 30 02:42:11.153136 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.153087 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8"] Oct 30 02:42:11.157601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.157568 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b93d51-6f4bdfb747-gr4c8"] Oct 30 02:42:11.205817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.205782 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9"] Oct 30 02:42:11.211915 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.211878 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b93d51-66c97d5bb6-2gpw9"] Oct 30 02:42:11.741031 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.740998 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69742b2d-68ad-4041-af3c-c80740eca721" path="/var/lib/kubelet/pods/69742b2d-68ad-4041-af3c-c80740eca721/volumes" Oct 30 02:42:11.741508 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.741493 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" path="/var/lib/kubelet/pods/8cdf999e-4537-4613-896b-7c9736e8136e/volumes" Oct 30 02:42:11.741894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:11.741882 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" path="/var/lib/kubelet/pods/fbd25775-c6d6-4dcd-b19e-5db7800b899c/volumes" Oct 30 02:42:16.703390 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:16.703352 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:42:20.062695 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.062658 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv"] Oct 30 02:42:20.063192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.062923 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" podUID="5c28a557-34b1-4cf2-8e2e-938dfd6b6857" containerName="maas-api" containerID="cri-o://04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f" gracePeriod=30 Oct 30 02:42:20.380027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.379978 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:42:20.466144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.465558 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca\") pod \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " Oct 30 02:42:20.466144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.465633 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8\" (UniqueName: \"kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8\") pod \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " Oct 30 02:42:20.466144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.465849 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls\") pod \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " Oct 30 02:42:20.466144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.465895 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs\") pod \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\" (UID: \"5c28a557-34b1-4cf2-8e2e-938dfd6b6857\") " Oct 30 02:42:20.474424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.474379 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca" pod "5c28a557-34b1-4cf2-8e2e-938dfd6b6857" (UID: "5c28a557-34b1-4cf2-8e2e-938dfd6b6857"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:20.487690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.487649 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8" pod "5c28a557-34b1-4cf2-8e2e-938dfd6b6857" (UID: "5c28a557-34b1-4cf2-8e2e-938dfd6b6857"). InnerVolumeSpecName "kube-api-access-nhhr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:20.507688 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.507654 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs" pod "5c28a557-34b1-4cf2-8e2e-938dfd6b6857" (UID: "5c28a557-34b1-4cf2-8e2e-938dfd6b6857"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:20.532439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.532377 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls" pod "5c28a557-34b1-4cf2-8e2e-938dfd6b6857" (UID: "5c28a557-34b1-4cf2-8e2e-938dfd6b6857"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:20.568328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.568288 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:20.568328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.568325 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:20.568328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.568337 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:20.568589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:20.568350 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhhr8\" (UniqueName: \"kubernetes.io/projected/5c28a557-34b1-4cf2-8e2e-938dfd6b6857-kube-api-access-nhhr8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:21.182216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.182184 6275 generic.go:356] "Generic (PLEG): container finished" podID="5c28a557-34b1-4cf2-8e2e-938dfd6b6857" containerID="04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f" exitCode=0 Oct 30 02:42:21.182631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.182233 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" Oct 30 02:42:21.182631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.182274 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" event={"ID":"5c28a557-34b1-4cf2-8e2e-938dfd6b6857","Type":"ContainerDied","Data":"04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f"} Oct 30 02:42:21.182631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.182330 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv" event={"ID":"5c28a557-34b1-4cf2-8e2e-938dfd6b6857","Type":"ContainerDied","Data":"30074e306b5c82cdf073bfdc805ad1a1a8f50babcd486d70165b6b411df9fed2"} Oct 30 02:42:21.182631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.182352 6275 scope.go:122] "RemoveContainer" containerID="04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f" Oct 30 02:42:21.192812 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.192796 6275 scope.go:122] "RemoveContainer" containerID="04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f" Oct 30 02:42:21.193069 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:21.193052 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f\": container with ID starting with 04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f not found: ID does not exist" containerID="04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f" Oct 30 02:42:21.193135 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.193078 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f"} err="failed to get container status \"04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f\": rpc error: code = NotFound desc = could not find container \"04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f\": container with ID starting with 04ff79a00e9bd4fb150c5569ae7ceb7d5fc6ef23486d1df90dfd0b6ed1ae095f not found: ID does not exist" Oct 30 02:42:21.230813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.230780 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv"] Oct 30 02:42:21.240841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.240813 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a00b8-68497f9f9b-kj4nv"] Oct 30 02:42:21.365956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.364999 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld"] Oct 30 02:42:21.365956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.365295 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" containerID="cri-o://accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0" gracePeriod=30 Oct 30 02:42:21.740417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:21.740340 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c28a557-34b1-4cf2-8e2e-938dfd6b6857" path="/var/lib/kubelet/pods/5c28a557-34b1-4cf2-8e2e-938dfd6b6857/volumes" Oct 30 02:42:24.566506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:24.566467 6275 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:42:24.566933 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:24.566525 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:42:26.656808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.656777 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 02:42:26.762934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.762893 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763009 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763047 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763088 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763184 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763238 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763281 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763357 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:26.763521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763369 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:26.763521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763383 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:26.763521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763391 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763441 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo\") pod \"d57544d9-f1dc-492f-aa75-d00704637f9b\" (UID: \"d57544d9-f1dc-492f-aa75-d00704637f9b\") " Oct 30 02:42:26.763779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763604 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:26.763779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.763676 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:26.764100 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.764075 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.764204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.764103 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.764204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.764151 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.764204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.764166 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57544d9-f1dc-492f-aa75-d00704637f9b-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.764204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.764179 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.766178 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.766113 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "kube-api-access-8nqq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:26.766300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.766193 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:26.766374 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.766300 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:26.768248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.768224 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo" pod "d57544d9-f1dc-492f-aa75-d00704637f9b" (UID: "d57544d9-f1dc-492f-aa75-d00704637f9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:42:26.865115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.865081 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.865115 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.865111 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nqq7\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-kube-api-access-8nqq7\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.865355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.865155 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:26.865355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:26.865165 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57544d9-f1dc-492f-aa75-d00704637f9b-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:27.236810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.236778 6275 generic.go:356] "Generic (PLEG): container finished" podID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerID="accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0" exitCode=0 Oct 30 02:42:27.236975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.236833 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" Oct 30 02:42:27.236975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.236867 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" event={"ID":"d57544d9-f1dc-492f-aa75-d00704637f9b","Type":"ContainerDied","Data":"accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0"} Oct 30 02:42:27.236975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.236921 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld" event={"ID":"d57544d9-f1dc-492f-aa75-d00704637f9b","Type":"ContainerDied","Data":"99946bec5d5f18461f7ce098b8ff087c16566d33cb7bca32ff6154db5ff70d10"} Oct 30 02:42:27.236975 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.236950 6275 scope.go:122] "RemoveContainer" containerID="accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0" Oct 30 02:42:27.248305 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.248286 6275 scope.go:122] "RemoveContainer" containerID="9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc" Oct 30 02:42:27.258035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.258010 6275 scope.go:122] "RemoveContainer" containerID="accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0" Oct 30 02:42:27.258389 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:27.258371 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0\": container with ID starting with accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0 not found: ID does not exist" containerID="accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0" Oct 30 02:42:27.258624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.258398 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0"} err="failed to get container status \"accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0\": rpc error: code = NotFound desc = could not find container \"accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0\": container with ID starting with accd77d0e62d9c30767add2a17bf610cf5ba6c63d3e725b633c76f9e92ad62b0 not found: ID does not exist" Oct 30 02:42:27.258624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.258416 6275 scope.go:122] "RemoveContainer" containerID="9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc" Oct 30 02:42:27.258717 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:27.258626 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc\": container with ID starting with 9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc not found: ID does not exist" containerID="9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc" Oct 30 02:42:27.258717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.258641 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc"} err="failed to get container status \"9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc\": rpc error: code = NotFound desc = could not find container \"9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc\": container with ID starting with 9e5d5dfcbfd5e47d8efc549b9755973bf98051889d83df1e9e5433f70f4205bc not found: ID does not exist" Oct 30 02:42:27.285604 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.285571 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld"] Oct 30 02:42:27.295725 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.295697 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-xw5ld"] Oct 30 02:42:27.502679 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.502588 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488"] Oct 30 02:42:27.502874 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.502831 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" containerID="cri-o://4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a" gracePeriod=30 Oct 30 02:42:27.742712 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:27.742678 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" path="/var/lib/kubelet/pods/d57544d9-f1dc-492f-aa75-d00704637f9b/volumes" Oct 30 02:42:28.584558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:28.584503 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:42:28.584745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:28.584588 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:42:31.232336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.232301 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh"] Oct 30 02:42:31.233036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233015 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" Oct 30 02:42:31.233036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233035 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233049 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233054 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233069 6275 cpu_manager.go:414] "containerMap: removing container" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerName="payload-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233074 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerName="payload-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233084 6275 cpu_manager.go:414] "containerMap: removing container" podUID="90615baf-1842-45de-b5ed-57f5a59d02f4" containerName="payload-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233090 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90615baf-1842-45de-b5ed-57f5a59d02f4" containerName="payload-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233106 6275 cpu_manager.go:414] "containerMap: removing container" podUID="69742b2d-68ad-4041-af3c-c80740eca721" containerName="maas-api" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233111 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69742b2d-68ad-4041-af3c-c80740eca721" containerName="maas-api" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233151 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5c28a557-34b1-4cf2-8e2e-938dfd6b6857" containerName="maas-api" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233156 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c28a557-34b1-4cf2-8e2e-938dfd6b6857" containerName="maas-api" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233167 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" containerName="payload-pre-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233171 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" containerName="payload-pre-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233179 6275 cpu_manager.go:414] "containerMap: removing container" podUID="05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" containerName="payload-pre-processing" Oct 30 02:42:31.233211 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233184 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" containerName="payload-pre-processing" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233277 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd25775-c6d6-4dcd-b19e-5db7800b899c" containerName="payload-processing" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233284 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="90615baf-1842-45de-b5ed-57f5a59d02f4" containerName="payload-processing" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233295 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="69742b2d-68ad-4041-af3c-c80740eca721" containerName="maas-api" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233303 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c28a557-34b1-4cf2-8e2e-938dfd6b6857" containerName="maas-api" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233310 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cdf999e-4537-4613-896b-7c9736e8136e" containerName="payload-pre-processing" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233316 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ad4b0f-1c8e-47b5-a4ee-a004016ceb8e" containerName="payload-pre-processing" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233324 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" Oct 30 02:42:31.233971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.233329 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57544d9-f1dc-492f-aa75-d00704637f9b" containerName="istio-proxy" Oct 30 02:42:31.238670 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.238649 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.255820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.255792 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh"] Oct 30 02:42:31.262713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.262681 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"openshift-service-ca.crt\"" Oct 30 02:42:31.262859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.262718 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"kube-root-ca.crt\"" Oct 30 02:42:31.262859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.262788 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"rate-test-model-01a9cd-kserve-self-signed-certs\"" Oct 30 02:42:31.262859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.262851 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"default-dockercfg-nscnx\"" Oct 30 02:42:31.322910 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.322874 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.323154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.322946 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.323154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.323052 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.323154 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.323096 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.323294 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.323256 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.323343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.323290 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425202 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425165 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425243 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425265 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425285 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425353 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425375 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425656 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425736 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425684 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.425736 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.425720 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.427772 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.427749 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.428252 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.428229 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.482146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.479867 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk\") pod \"rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.554329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.554293 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:31.863235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:31.863205 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh"] Oct 30 02:42:31.865551 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:42:31.865523 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf70cf0e_c1ec_42dc_a4a7_244d89d835ac.slice/crio-7f7de5fe700a3ea84a5ecb3d100bd2c9754cb76dc93bd8642d2f24d4b7a165f7 WatchSource:0}: Error finding container 7f7de5fe700a3ea84a5ecb3d100bd2c9754cb76dc93bd8642d2f24d4b7a165f7: Status 404 returned error can't find the container with id 7f7de5fe700a3ea84a5ecb3d100bd2c9754cb76dc93bd8642d2f24d4b7a165f7 Oct 30 02:42:32.275071 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.275025 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" event={"ID":"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac","Type":"ContainerStarted","Data":"7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3"} Oct 30 02:42:32.275557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.275079 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" event={"ID":"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac","Type":"ContainerStarted","Data":"7f7de5fe700a3ea84a5ecb3d100bd2c9754cb76dc93bd8642d2f24d4b7a165f7"} Oct 30 02:42:32.320967 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.320910 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" podStartSLOduration=1.320894205 podStartE2EDuration="1.320894205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:32.317761569 +0000 UTC m=+4380.709569203" watchObservedRunningTime="2026-10-30 02:42:32.320894205 +0000 UTC m=+4380.712701838" Oct 30 02:42:32.554631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.554587 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:32.780729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.780698 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:42:32.967683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967594 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.967683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967633 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.967683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967674 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.967941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967853 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.967941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967882 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:32.967941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967891 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.968149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967972 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:32.968149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.967999 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.968149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968053 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.968149 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968144 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.968373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968174 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert\") pod \"0cc52375-b571-4ab9-bba6-24ccf15db366\" (UID: \"0cc52375-b571-4ab9-bba6-24ccf15db366\") " Oct 30 02:42:32.968373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968203 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:32.968373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968324 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:32.968633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968599 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:32.968826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968809 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:32.968899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968833 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:32.968899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968847 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc52375-b571-4ab9-bba6-24ccf15db366-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:32.968899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968862 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:32.968899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.968876 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:32.970205 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.970185 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:42:32.970318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.970286 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:32.970478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.970463 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:42:32.970588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:32.970571 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm" pod "0cc52375-b571-4ab9-bba6-24ccf15db366" (UID: "0cc52375-b571-4ab9-bba6-24ccf15db366"). InnerVolumeSpecName "kube-api-access-j4hcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:33.069822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.069791 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hcm\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-kube-api-access-j4hcm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:33.069822 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.069822 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:33.070033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.069832 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:33.070033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.069839 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc52375-b571-4ab9-bba6-24ccf15db366-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:42:33.284622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.284587 6275 generic.go:356] "Generic (PLEG): container finished" podID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerID="4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a" exitCode=0 Oct 30 02:42:33.285077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.284650 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" Oct 30 02:42:33.285077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.284672 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" event={"ID":"0cc52375-b571-4ab9-bba6-24ccf15db366","Type":"ContainerDied","Data":"4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a"} Oct 30 02:42:33.285077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.284761 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488" event={"ID":"0cc52375-b571-4ab9-bba6-24ccf15db366","Type":"ContainerDied","Data":"447f6fdb3526176620ce4c4f035c8d35cd8e6b3a8ba45e96425230f0e3ac6e07"} Oct 30 02:42:33.285077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.284787 6275 scope.go:122] "RemoveContainer" containerID="4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a" Oct 30 02:42:33.295786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.295768 6275 scope.go:122] "RemoveContainer" containerID="3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e" Oct 30 02:42:33.305658 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.305639 6275 scope.go:122] "RemoveContainer" containerID="4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a" Oct 30 02:42:33.305915 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:33.305894 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a\": container with ID starting with 4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a not found: ID does not exist" containerID="4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a" Oct 30 02:42:33.305970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.305928 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a"} err="failed to get container status \"4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a\": rpc error: code = NotFound desc = could not find container \"4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a\": container with ID starting with 4db7578535f1e9262a1e925331f8a0fd06173447a0ab760f8f7ec00c56616e9a not found: ID does not exist" Oct 30 02:42:33.305970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.305949 6275 scope.go:122] "RemoveContainer" containerID="3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e" Oct 30 02:42:33.306241 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:33.306223 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e\": container with ID starting with 3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e not found: ID does not exist" containerID="3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e" Oct 30 02:42:33.306301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.306250 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e"} err="failed to get container status \"3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e\": rpc error: code = NotFound desc = could not find container \"3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e\": container with ID starting with 3b78d5ec03cc58078fa69b8b591bf6e30ae83c015c5598318ce8a79090f22d1e not found: ID does not exist" Oct 30 02:42:33.331540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.331509 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488"] Oct 30 02:42:33.342195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.342162 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3a00b8-openshift-default-6b6cb8c5cc-xk488"] Oct 30 02:42:33.741880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:33.741764 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" path="/var/lib/kubelet/pods/0cc52375-b571-4ab9-bba6-24ccf15db366/volumes" Oct 30 02:42:41.554836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:41.554794 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:41.567489 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:41.567462 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:42.354957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:42:42.354925 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:42:46.700008 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:42:46.699964 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:43:07.118993 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:07.118957 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm"] Oct 30 02:43:16.702070 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:43:16.701989 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:43:22.140866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.140797 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" containerID="cri-o://ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75" gracePeriod=45 Oct 30 02:43:22.408829 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.408745 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 02:43:22.498500 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498468 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz\" (UniqueName: \"kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz\") pod \"2c431280-4854-4765-8c94-9cee52957245\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " Oct 30 02:43:22.498709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498522 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home\") pod \"2c431280-4854-4765-8c94-9cee52957245\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " Oct 30 02:43:22.498709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498603 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm\") pod \"2c431280-4854-4765-8c94-9cee52957245\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " Oct 30 02:43:22.498709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498690 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir\") pod \"2c431280-4854-4765-8c94-9cee52957245\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " Oct 30 02:43:22.498908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498737 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs\") pod \"2c431280-4854-4765-8c94-9cee52957245\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " Oct 30 02:43:22.498908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498771 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache\") pod \"2c431280-4854-4765-8c94-9cee52957245\" (UID: \"2c431280-4854-4765-8c94-9cee52957245\") " Oct 30 02:43:22.498908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498834 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home" pod "2c431280-4854-4765-8c94-9cee52957245" (UID: "2c431280-4854-4765-8c94-9cee52957245"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.499094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.498957 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir" pod "2c431280-4854-4765-8c94-9cee52957245" (UID: "2c431280-4854-4765-8c94-9cee52957245"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.499206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.499181 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache" pod "2c431280-4854-4765-8c94-9cee52957245" (UID: "2c431280-4854-4765-8c94-9cee52957245"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.499333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.499317 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.499398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.499336 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.499398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.499347 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.500845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.500821 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm" pod "2c431280-4854-4765-8c94-9cee52957245" (UID: "2c431280-4854-4765-8c94-9cee52957245"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.501090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.501069 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz" pod "2c431280-4854-4765-8c94-9cee52957245" (UID: "2c431280-4854-4765-8c94-9cee52957245"). InnerVolumeSpecName "kube-api-access-6rwgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:22.501224 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.501092 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs" pod "2c431280-4854-4765-8c94-9cee52957245" (UID: "2c431280-4854-4765-8c94-9cee52957245"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:22.600849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.600813 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c431280-4854-4765-8c94-9cee52957245-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.600849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.600850 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rwgz\" (UniqueName: \"kubernetes.io/projected/2c431280-4854-4765-8c94-9cee52957245-kube-api-access-6rwgz\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.601062 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.600865 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c431280-4854-4765-8c94-9cee52957245-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.603612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.603577 6275 generic.go:356] "Generic (PLEG): container finished" podID="2c431280-4854-4765-8c94-9cee52957245" containerID="ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75" exitCode=0 Oct 30 02:43:22.603768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.603622 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" Oct 30 02:43:22.603768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.603618 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" event={"ID":"2c431280-4854-4765-8c94-9cee52957245","Type":"ContainerDied","Data":"ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75"} Oct 30 02:43:22.603768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.603733 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm" event={"ID":"2c431280-4854-4765-8c94-9cee52957245","Type":"ContainerDied","Data":"80378f94c0c9291ffca7e5b84fba6f69d6405857ba0be92f48c8eef32d4de475"} Oct 30 02:43:22.603768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.603754 6275 scope.go:122] "RemoveContainer" containerID="ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75" Oct 30 02:43:22.616831 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.616804 6275 scope.go:122] "RemoveContainer" containerID="ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75" Oct 30 02:43:22.617177 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:43:22.617152 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75\": container with ID starting with ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75 not found: ID does not exist" containerID="ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75" Oct 30 02:43:22.617304 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.617184 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75"} err="failed to get container status \"ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75\": rpc error: code = NotFound desc = could not find container \"ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75\": container with ID starting with ebfce913c128466d95baab2473c867527fd32c78530f28363dfd57747a1c7a75 not found: ID does not exist" Oct 30 02:43:22.650876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.650838 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm"] Oct 30 02:43:22.656505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:22.656476 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-2-w1-f6f4f0-kserve-7884565899-4ghkm"] Oct 30 02:43:23.740835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:23.740800 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c431280-4854-4765-8c94-9cee52957245" path="/var/lib/kubelet/pods/2c431280-4854-4765-8c94-9cee52957245/volumes" Oct 30 02:43:44.634431 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.634392 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj"] Oct 30 02:43:44.636309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.636277 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" Oct 30 02:43:44.636499 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.636484 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" Oct 30 02:43:44.636598 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.636587 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" Oct 30 02:43:44.636681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.636671 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" Oct 30 02:43:44.636804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.636794 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" Oct 30 02:43:44.636884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.636874 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" Oct 30 02:43:44.650339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.650300 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" Oct 30 02:43:44.650667 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.650629 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c431280-4854-4765-8c94-9cee52957245" containerName="main" Oct 30 02:43:44.650871 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.650853 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc52375-b571-4ab9-bba6-24ccf15db366" containerName="istio-proxy" Oct 30 02:43:44.663039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.663004 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.673176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.673142 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj"] Oct 30 02:43:44.676841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.676811 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f6f4f0-openshift-default-dockercfg-jl28w\"" Oct 30 02:43:44.708141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708100 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708329 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708293 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708451 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708345 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708509 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708463 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708585 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708699 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708680 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708767 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708731 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708783 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mrg\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.708913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.708825 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.809784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.809741 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.809970 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.809929 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810024 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.809991 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810033 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mrg\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810065 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810395 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810437 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810552 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810594 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810638 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810825 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810678 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810825 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810679 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.810971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810949 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.811044 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.810947 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.812419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.812391 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.812664 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.812645 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.831531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.831494 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.831660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.831621 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mrg\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:44.986280 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:44.986193 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:45.311110 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.311080 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj"] Oct 30 02:43:45.314877 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:43:45.314849 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b08d430_67c3_4733_8845_3a1da22a3d68.slice/crio-60f2d912141c95d89b059769966333a43f3106a959edfc1855496ad4fc3a2430 WatchSource:0}: Error finding container 60f2d912141c95d89b059769966333a43f3106a959edfc1855496ad4fc3a2430: Status 404 returned error can't find the container with id 60f2d912141c95d89b059769966333a43f3106a959edfc1855496ad4fc3a2430 Oct 30 02:43:45.318047 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.318002 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:43:45.318157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.318100 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:43:45.318212 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.318184 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:43:45.756417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.756327 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" event={"ID":"5b08d430-67c3-4733-8845-3a1da22a3d68","Type":"ContainerStarted","Data":"f26c390936bb1e685841f4a6f1ba2c3484ccd5aeff92f583cc58c04411a74055"} Oct 30 02:43:45.756417 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.756392 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" event={"ID":"5b08d430-67c3-4733-8845-3a1da22a3d68","Type":"ContainerStarted","Data":"60f2d912141c95d89b059769966333a43f3106a959edfc1855496ad4fc3a2430"} Oct 30 02:43:45.801608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.801548 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podStartSLOduration=1.801534075 podStartE2EDuration="1.801534075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:43:45.799837218 +0000 UTC m=+4454.191644852" watchObservedRunningTime="2026-10-30 02:43:45.801534075 +0000 UTC m=+4454.193341708" Oct 30 02:43:45.987398 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:45.986816 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:46.697055 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:43:46.697011 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:43:46.988467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:46.988387 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 02:43:46.988467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:46.988442 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 02:43:47.986581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:47.986543 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 02:43:47.986770 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:47.986605 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 02:43:48.987333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:48.987299 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 02:43:48.987748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:48.987355 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 02:43:49.987013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:49.986976 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 02:43:49.987206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:49.987031 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 02:43:50.987275 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:50.987238 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 02:43:50.987680 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:50.987300 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 02:43:51.986683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:51.986643 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 02:43:51.986869 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:51.986721 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 02:43:53.988157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:53.988094 6275 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:43:53.988779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:53.988185 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:43:54.007362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:54.007329 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:54.007762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:54.007737 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:54.008910 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:54.008885 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:43:58.657152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:58.647625 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj"] Oct 30 02:43:58.657152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:43:58.647841 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" containerID="cri-o://f26c390936bb1e685841f4a6f1ba2c3484ccd5aeff92f583cc58c04411a74055" gracePeriod=30 Oct 30 02:44:03.904917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:03.904882 6275 generic.go:356] "Generic (PLEG): container finished" podID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerID="f26c390936bb1e685841f4a6f1ba2c3484ccd5aeff92f583cc58c04411a74055" exitCode=0 Oct 30 02:44:03.905355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:03.904956 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" event={"ID":"5b08d430-67c3-4733-8845-3a1da22a3d68","Type":"ContainerDied","Data":"f26c390936bb1e685841f4a6f1ba2c3484ccd5aeff92f583cc58c04411a74055"} Oct 30 02:44:04.351459 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.351434 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:44:04.496309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496221 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496320 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496377 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496410 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496465 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496492 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496500 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.496593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496571 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.497028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496649 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.497028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496696 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket\") pod \"5b08d430-67c3-4733-8845-3a1da22a3d68\" (UID: \"5b08d430-67c3-4733-8845-3a1da22a3d68\") " Oct 30 02:44:04.497028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496731 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.497028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496885 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.497028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.496910 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:44:04.497350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.497102 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.497440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.497418 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.497515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.497445 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.497515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.497461 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b08d430-67c3-4733-8845-3a1da22a3d68-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.497515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.497475 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.497515 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.497490 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.498739 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.498710 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "kube-api-access-78mrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:04.498863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.498758 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.498863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.498789 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:44:04.498863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.498830 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token" pod "5b08d430-67c3-4733-8845-3a1da22a3d68" (UID: "5b08d430-67c3-4733-8845-3a1da22a3d68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:04.598991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.598957 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.598991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.598988 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mrg\" (UniqueName: \"kubernetes.io/projected/5b08d430-67c3-4733-8845-3a1da22a3d68-kube-api-access-78mrg\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.598991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.598998 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.599259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.599007 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b08d430-67c3-4733-8845-3a1da22a3d68-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.912113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.912081 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" Oct 30 02:44:04.912113 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.912105 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj" event={"ID":"5b08d430-67c3-4733-8845-3a1da22a3d68","Type":"ContainerDied","Data":"60f2d912141c95d89b059769966333a43f3106a959edfc1855496ad4fc3a2430"} Oct 30 02:44:04.912650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.912176 6275 scope.go:122] "RemoveContainer" containerID="f26c390936bb1e685841f4a6f1ba2c3484ccd5aeff92f583cc58c04411a74055" Oct 30 02:44:04.962467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.962433 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj"] Oct 30 02:44:04.977727 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:04.977687 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-tkqwj"] Oct 30 02:44:05.739833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:05.739795 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" path="/var/lib/kubelet/pods/5b08d430-67c3-4733-8845-3a1da22a3d68/volumes" Oct 30 02:44:16.696780 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:44:16.696738 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:44:32.680065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:32.680032 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:44:32.690372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:44:32.690349 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:44:46.722196 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:44:46.718928 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:45:00.343656 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.343615 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc"] Oct 30 02:45:00.345213 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.345187 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" Oct 30 02:45:00.345388 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.345375 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" Oct 30 02:45:00.345639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.345617 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b08d430-67c3-4733-8845-3a1da22a3d68" containerName="istio-proxy" Oct 30 02:45:00.353608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.353579 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4"] Oct 30 02:45:00.353807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.353696 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.358681 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.358659 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.367935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.367907 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc"] Oct 30 02:45:00.374063 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.374037 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:45:00.396535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.396503 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4"] Oct 30 02:45:00.460791 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.460750 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7l7s\" (UniqueName: \"kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.460791 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.460789 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgvv\" (UniqueName: \"kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.461087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.461066 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.461160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.461109 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.562324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.562293 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.562514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.562456 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7l7s\" (UniqueName: \"kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.562514 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.562492 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgvv\" (UniqueName: \"kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.562706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.562691 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.562941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.562921 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.563317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.563298 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.584353 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.584324 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7l7s\" (UniqueName: \"kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.584486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.584375 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgvv\" (UniqueName: \"kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.677497 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.677416 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:00.689803 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.689775 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:00.990545 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:00.990514 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc"] Oct 30 02:45:00.993787 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:45:00.993748 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d2ac4c_10bf_47e7_bff0_254447ab0b62.slice/crio-9138eae92e7f14ef29e4d820fd36131442a52aa5ab1e06539a119c4fbaf92031 WatchSource:0}: Error finding container 9138eae92e7f14ef29e4d820fd36131442a52aa5ab1e06539a119c4fbaf92031: Status 404 returned error can't find the container with id 9138eae92e7f14ef29e4d820fd36131442a52aa5ab1e06539a119c4fbaf92031 Oct 30 02:45:01.009793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:01.009731 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4"] Oct 30 02:45:01.012420 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:45:01.012378 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d45a619_bd39_48ff_84d2_c23e59556508.slice/crio-e6ad261de92e798bfc0b97ab07a65172430f8fa9e71925f3ecf7e535b887461c WatchSource:0}: Error finding container e6ad261de92e798bfc0b97ab07a65172430f8fa9e71925f3ecf7e535b887461c: Status 404 returned error can't find the container with id e6ad261de92e798bfc0b97ab07a65172430f8fa9e71925f3ecf7e535b887461c Oct 30 02:45:01.324640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:01.324599 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" event={"ID":"d6d2ac4c-10bf-47e7-bff0-254447ab0b62","Type":"ContainerStarted","Data":"9138eae92e7f14ef29e4d820fd36131442a52aa5ab1e06539a119c4fbaf92031"} Oct 30 02:45:01.326080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:01.326053 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerStarted","Data":"e6ad261de92e798bfc0b97ab07a65172430f8fa9e71925f3ecf7e535b887461c"} Oct 30 02:45:02.335418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:02.335382 6275 generic.go:356] "Generic (PLEG): container finished" podID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerID="a5f4c21956cf449c2be5cf071588bf04c26bf15aa765c3092aeaa0d52b620f28" exitCode=60 Oct 30 02:45:02.335864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:02.335467 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" event={"ID":"d6d2ac4c-10bf-47e7-bff0-254447ab0b62","Type":"ContainerDied","Data":"a5f4c21956cf449c2be5cf071588bf04c26bf15aa765c3092aeaa0d52b620f28"} Oct 30 02:45:02.335864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:02.335693 6275 scope.go:122] "RemoveContainer" containerID="a5f4c21956cf449c2be5cf071588bf04c26bf15aa765c3092aeaa0d52b620f28" Oct 30 02:45:02.337228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:02.337197 6275 generic.go:356] "Generic (PLEG): container finished" podID="1d45a619-bd39-48ff-84d2-c23e59556508" containerID="28c87e7a33158759c90a2e31fffc2eafb160e1bb06bee044bc31b759d445eb55" exitCode=60 Oct 30 02:45:02.337396 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:02.337237 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerDied","Data":"28c87e7a33158759c90a2e31fffc2eafb160e1bb06bee044bc31b759d445eb55"} Oct 30 02:45:02.337483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:02.337468 6275 scope.go:122] "RemoveContainer" containerID="28c87e7a33158759c90a2e31fffc2eafb160e1bb06bee044bc31b759d445eb55" Oct 30 02:45:03.347985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:03.347949 6275 generic.go:356] "Generic (PLEG): container finished" podID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerID="70cdd421bb9bbaca500bd1269963d4be329ba78ecdbb9fc6c18d31eb827d85b0" exitCode=60 Oct 30 02:45:03.348413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:03.348269 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" event={"ID":"d6d2ac4c-10bf-47e7-bff0-254447ab0b62","Type":"ContainerDied","Data":"70cdd421bb9bbaca500bd1269963d4be329ba78ecdbb9fc6c18d31eb827d85b0"} Oct 30 02:45:03.348413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:03.348368 6275 scope.go:122] "RemoveContainer" containerID="a5f4c21956cf449c2be5cf071588bf04c26bf15aa765c3092aeaa0d52b620f28" Oct 30 02:45:03.348544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:03.348450 6275 scope.go:122] "RemoveContainer" containerID="70cdd421bb9bbaca500bd1269963d4be329ba78ecdbb9fc6c18d31eb827d85b0" Oct 30 02:45:03.348796 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:03.348774 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc_odh-ai-gateway-infra(d6d2ac4c-10bf-47e7-bff0-254447ab0b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" Oct 30 02:45:03.351047 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:03.351024 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerStarted","Data":"80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574"} Oct 30 02:45:03.427778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:03.427711 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" podStartSLOduration=3.427692989 podStartE2EDuration="3.427692989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:03.427314807 +0000 UTC m=+4531.819122446" watchObservedRunningTime="2026-10-30 02:45:03.427692989 +0000 UTC m=+4531.819500623" Oct 30 02:45:04.363141 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:04.361803 6275 scope.go:122] "RemoveContainer" containerID="70cdd421bb9bbaca500bd1269963d4be329ba78ecdbb9fc6c18d31eb827d85b0" Oct 30 02:45:04.363141 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:04.362241 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc_odh-ai-gateway-infra(d6d2ac4c-10bf-47e7-bff0-254447ab0b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" Oct 30 02:45:04.363795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:04.363770 6275 generic.go:356] "Generic (PLEG): container finished" podID="1d45a619-bd39-48ff-84d2-c23e59556508" containerID="80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574" exitCode=60 Oct 30 02:45:04.363946 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:04.363860 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerDied","Data":"80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574"} Oct 30 02:45:04.364017 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:04.363945 6275 scope.go:122] "RemoveContainer" containerID="28c87e7a33158759c90a2e31fffc2eafb160e1bb06bee044bc31b759d445eb55" Oct 30 02:45:04.364150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:04.364114 6275 scope.go:122] "RemoveContainer" containerID="80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574" Oct 30 02:45:04.364442 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:04.364420 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4_odh-ai-gateway-infra(1d45a619-bd39-48ff-84d2-c23e59556508)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" Oct 30 02:45:05.373389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:05.373362 6275 scope.go:122] "RemoveContainer" containerID="80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574" Oct 30 02:45:05.373892 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:05.373649 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4_odh-ai-gateway-infra(1d45a619-bd39-48ff-84d2-c23e59556508)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" Oct 30 02:45:07.536146 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.536086 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx"] Oct 30 02:45:07.540379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.540360 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.562496 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.562464 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:45:07.577199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.577159 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx"] Oct 30 02:45:07.663754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.663715 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m896z\" (UniqueName: \"kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.663963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.663846 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.663963 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.663913 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.765363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.765328 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m896z\" (UniqueName: \"kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.765555 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.765390 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.765555 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.765424 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.766153 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.766116 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.767902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.767864 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.787269 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.787210 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m896z\" (UniqueName: \"kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z\") pod \"maas-api-revoke-sub-7ab41952-jdjkx\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:07.854889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:07.854859 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:08.161581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:08.161497 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx"] Oct 30 02:45:08.170290 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:45:08.170259 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82f90c0_f1a2_46b7_8dd5_117b35481b71.slice/crio-769a6699666fef5fa618375104c364ea187f5bee2212449b367ed7c979ce2ecd WatchSource:0}: Error finding container 769a6699666fef5fa618375104c364ea187f5bee2212449b367ed7c979ce2ecd: Status 404 returned error can't find the container with id 769a6699666fef5fa618375104c364ea187f5bee2212449b367ed7c979ce2ecd Oct 30 02:45:08.455766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:08.455672 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" event={"ID":"e82f90c0-f1a2-46b7-8dd5-117b35481b71","Type":"ContainerStarted","Data":"769a6699666fef5fa618375104c364ea187f5bee2212449b367ed7c979ce2ecd"} Oct 30 02:45:09.463711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:09.463630 6275 generic.go:356] "Generic (PLEG): container finished" podID="e82f90c0-f1a2-46b7-8dd5-117b35481b71" containerID="9cfc6cb4bd68dac18804f2a98637a975afd7944a6540f72d8336c7a5d9a81007" exitCode=0 Oct 30 02:45:09.464107 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:09.463712 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" event={"ID":"e82f90c0-f1a2-46b7-8dd5-117b35481b71","Type":"ContainerDied","Data":"9cfc6cb4bd68dac18804f2a98637a975afd7944a6540f72d8336c7a5d9a81007"} Oct 30 02:45:10.620602 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.620572 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:10.718109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.718078 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca\") pod \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " Oct 30 02:45:10.718310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.718212 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca\") pod \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " Oct 30 02:45:10.718384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.718336 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z\" (UniqueName: \"kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z\") pod \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\" (UID: \"e82f90c0-f1a2-46b7-8dd5-117b35481b71\") " Oct 30 02:45:10.718616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.718587 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca" pod "e82f90c0-f1a2-46b7-8dd5-117b35481b71" (UID: "e82f90c0-f1a2-46b7-8dd5-117b35481b71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:10.718800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.718784 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-openshift-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:10.720334 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.720307 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca" pod "e82f90c0-f1a2-46b7-8dd5-117b35481b71" (UID: "e82f90c0-f1a2-46b7-8dd5-117b35481b71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:10.720449 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.720382 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z" pod "e82f90c0-f1a2-46b7-8dd5-117b35481b71" (UID: "e82f90c0-f1a2-46b7-8dd5-117b35481b71"). InnerVolumeSpecName "kube-api-access-m896z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:10.820522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.820479 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m896z\" (UniqueName: \"kubernetes.io/projected/e82f90c0-f1a2-46b7-8dd5-117b35481b71-kube-api-access-m896z\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:10.820522 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:10.820521 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82f90c0-f1a2-46b7-8dd5-117b35481b71-maas-api-xks-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:11.478349 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:11.478314 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" Oct 30 02:45:11.478530 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:11.478347 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx" event={"ID":"e82f90c0-f1a2-46b7-8dd5-117b35481b71","Type":"ContainerDied","Data":"769a6699666fef5fa618375104c364ea187f5bee2212449b367ed7c979ce2ecd"} Oct 30 02:45:11.478530 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:11.478400 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="769a6699666fef5fa618375104c364ea187f5bee2212449b367ed7c979ce2ecd" Oct 30 02:45:15.253440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:15.253402 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p"] Oct 30 02:45:16.697440 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:16.697400 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:45:17.731919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:17.731877 6275 scope.go:122] "RemoveContainer" containerID="70cdd421bb9bbaca500bd1269963d4be329ba78ecdbb9fc6c18d31eb827d85b0" Oct 30 02:45:17.732392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:17.732014 6275 scope.go:122] "RemoveContainer" containerID="80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574" Oct 30 02:45:18.524840 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:18.524799 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" event={"ID":"d6d2ac4c-10bf-47e7-bff0-254447ab0b62","Type":"ContainerStarted","Data":"755ba8a60de2872d0b0410e387eeb0b0c8e2c74b9139472f65cbbdb7058d5563"} Oct 30 02:45:18.526987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:18.526955 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerStarted","Data":"5158eef69fdf6026c8fcddaf3e4ee3b3c905c113fb0840c965bf7a9f79aa238a"} Oct 30 02:45:18.562250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:18.562187 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" podStartSLOduration=18.562174099 podStartE2EDuration="18.562174099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:18.561749144 +0000 UTC m=+4546.953556778" watchObservedRunningTime="2026-10-30 02:45:18.562174099 +0000 UTC m=+4546.953981732" Oct 30 02:45:19.534818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.534782 6275 generic.go:356] "Generic (PLEG): container finished" podID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerID="755ba8a60de2872d0b0410e387eeb0b0c8e2c74b9139472f65cbbdb7058d5563" exitCode=60 Oct 30 02:45:19.535290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.534851 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" event={"ID":"d6d2ac4c-10bf-47e7-bff0-254447ab0b62","Type":"ContainerDied","Data":"755ba8a60de2872d0b0410e387eeb0b0c8e2c74b9139472f65cbbdb7058d5563"} Oct 30 02:45:19.535290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.534911 6275 scope.go:122] "RemoveContainer" containerID="70cdd421bb9bbaca500bd1269963d4be329ba78ecdbb9fc6c18d31eb827d85b0" Oct 30 02:45:19.535408 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.535351 6275 scope.go:122] "RemoveContainer" containerID="755ba8a60de2872d0b0410e387eeb0b0c8e2c74b9139472f65cbbdb7058d5563" Oct 30 02:45:19.535719 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:19.535686 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc_odh-ai-gateway-infra(d6d2ac4c-10bf-47e7-bff0-254447ab0b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" Oct 30 02:45:19.537005 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.536985 6275 generic.go:356] "Generic (PLEG): container finished" podID="1d45a619-bd39-48ff-84d2-c23e59556508" containerID="5158eef69fdf6026c8fcddaf3e4ee3b3c905c113fb0840c965bf7a9f79aa238a" exitCode=60 Oct 30 02:45:19.537090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.537062 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerDied","Data":"5158eef69fdf6026c8fcddaf3e4ee3b3c905c113fb0840c965bf7a9f79aa238a"} Oct 30 02:45:19.537426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.537414 6275 scope.go:122] "RemoveContainer" containerID="5158eef69fdf6026c8fcddaf3e4ee3b3c905c113fb0840c965bf7a9f79aa238a" Oct 30 02:45:19.537671 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:19.537653 6275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4_odh-ai-gateway-infra(1d45a619-bd39-48ff-84d2-c23e59556508)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" Oct 30 02:45:19.547684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.547664 6275 scope.go:122] "RemoveContainer" containerID="80e0e2af5b26cb4a711526cff4f135861a5f112a957910a53c31455f91c19574" Oct 30 02:45:19.573584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.573550 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc"] Oct 30 02:45:19.609851 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:19.609814 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4"] Oct 30 02:45:20.726243 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.726218 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:20.755568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.755545 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:20.864836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.864745 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv\" (UniqueName: \"kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv\") pod \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " Oct 30 02:45:20.864836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.864804 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s\" (UniqueName: \"kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s\") pod \"1d45a619-bd39-48ff-84d2-c23e59556508\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " Oct 30 02:45:20.865206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.864887 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca\") pod \"1d45a619-bd39-48ff-84d2-c23e59556508\" (UID: \"1d45a619-bd39-48ff-84d2-c23e59556508\") " Oct 30 02:45:20.865206 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.864922 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca\") pod \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\" (UID: \"d6d2ac4c-10bf-47e7-bff0-254447ab0b62\") " Oct 30 02:45:20.865358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.865329 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca" pod "1d45a619-bd39-48ff-84d2-c23e59556508" (UID: "1d45a619-bd39-48ff-84d2-c23e59556508"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:20.865400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.865352 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca" pod "d6d2ac4c-10bf-47e7-bff0-254447ab0b62" (UID: "d6d2ac4c-10bf-47e7-bff0-254447ab0b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:20.866958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.866928 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv" pod "d6d2ac4c-10bf-47e7-bff0-254447ab0b62" (UID: "d6d2ac4c-10bf-47e7-bff0-254447ab0b62"). InnerVolumeSpecName "kube-api-access-lwgvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:20.867068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.866964 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s" pod "1d45a619-bd39-48ff-84d2-c23e59556508" (UID: "1d45a619-bd39-48ff-84d2-c23e59556508"). InnerVolumeSpecName "kube-api-access-v7l7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:20.966902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.966871 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwgvv\" (UniqueName: \"kubernetes.io/projected/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-kube-api-access-lwgvv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.966902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.966900 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7l7s\" (UniqueName: \"kubernetes.io/projected/1d45a619-bd39-48ff-84d2-c23e59556508-kube-api-access-v7l7s\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.966902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.966911 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d45a619-bd39-48ff-84d2-c23e59556508-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.967176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:20.966921 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d2ac4c-10bf-47e7-bff0-254447ab0b62-maas-api-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.064562 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.064527 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx"] Oct 30 02:45:21.554277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.554246 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" Oct 30 02:45:21.554277 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.554268 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4" event={"ID":"1d45a619-bd39-48ff-84d2-c23e59556508","Type":"ContainerDied","Data":"e6ad261de92e798bfc0b97ab07a65172430f8fa9e71925f3ecf7e535b887461c"} Oct 30 02:45:21.554557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.554344 6275 scope.go:122] "RemoveContainer" containerID="5158eef69fdf6026c8fcddaf3e4ee3b3c905c113fb0840c965bf7a9f79aa238a" Oct 30 02:45:21.556086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.556055 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" event={"ID":"d6d2ac4c-10bf-47e7-bff0-254447ab0b62","Type":"ContainerDied","Data":"9138eae92e7f14ef29e4d820fd36131442a52aa5ab1e06539a119c4fbaf92031"} Oct 30 02:45:21.556600 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.556538 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc" Oct 30 02:45:21.567648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.567569 6275 scope.go:122] "RemoveContainer" containerID="755ba8a60de2872d0b0410e387eeb0b0c8e2c74b9139472f65cbbdb7058d5563" Oct 30 02:45:21.619017 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.618986 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4"] Oct 30 02:45:21.624384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.624359 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888805-vghm4"] Oct 30 02:45:21.729962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.729925 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc"] Oct 30 02:45:21.760040 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.760006 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" path="/var/lib/kubelet/pods/1d45a619-bd39-48ff-84d2-c23e59556508/volumes" Oct 30 02:45:21.760428 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:21.760386 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888805-dx4jc"] Oct 30 02:45:22.733632 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:22.733595 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh"] Oct 30 02:45:23.739719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:23.739687 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" path="/var/lib/kubelet/pods/d6d2ac4c-10bf-47e7-bff0-254447ab0b62/volumes" Oct 30 02:45:30.276311 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.276268 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" podUID="1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" containerName="main" containerID="cri-o://75caa579ea80ae1a36e5710af739089d0430fff3d655cbd0a8a68f00f2c081ba" gracePeriod=45 Oct 30 02:45:30.664652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.664443 6275 generic.go:356] "Generic (PLEG): container finished" podID="1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" containerID="75caa579ea80ae1a36e5710af739089d0430fff3d655cbd0a8a68f00f2c081ba" exitCode=0 Oct 30 02:45:30.664652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.664592 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" event={"ID":"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a","Type":"ContainerDied","Data":"75caa579ea80ae1a36e5710af739089d0430fff3d655cbd0a8a68f00f2c081ba"} Oct 30 02:45:30.744245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.744212 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:45:30.809652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.809624 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs\") pod \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " Oct 30 02:45:30.809652 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.809666 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx\" (UniqueName: \"kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx\") pod \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " Oct 30 02:45:30.809921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.809703 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm\") pod \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " Oct 30 02:45:30.809921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.809812 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home\") pod \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " Oct 30 02:45:30.809921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.809886 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache\") pod \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " Oct 30 02:45:30.810086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.809973 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir\") pod \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\" (UID: \"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a\") " Oct 30 02:45:30.811169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.810316 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home" pod "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" (UID: "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:30.811169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.810323 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache" pod "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" (UID: "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:30.811169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.810792 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:30.811169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.810814 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:30.811169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.811136 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir" pod "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" (UID: "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:30.812598 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.812577 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm" pod "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" (UID: "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:30.812867 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.812803 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs" pod "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" (UID: "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:30.817013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.816988 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx" pod "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" (UID: "1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a"). InnerVolumeSpecName "kube-api-access-9pkbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:30.869610 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.869572 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2"] Oct 30 02:45:30.904150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.904100 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7"] Oct 30 02:45:30.912543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.912506 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:30.912543 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.912541 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:30.912753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.912554 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkbx\" (UniqueName: \"kubernetes.io/projected/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-kube-api-access-9pkbx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:30.912753 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:30.912566 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:31.672714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:31.672680 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" Oct 30 02:45:31.673221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:31.672712 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p" event={"ID":"1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a","Type":"ContainerDied","Data":"6c6f033f3ea411a6c7143776812cc0db956059e7d67723e6d80160720b82e355"} Oct 30 02:45:31.673221 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:31.672755 6275 scope.go:122] "RemoveContainer" containerID="75caa579ea80ae1a36e5710af739089d0430fff3d655cbd0a8a68f00f2c081ba" Oct 30 02:45:31.722554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:31.722527 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p"] Oct 30 02:45:31.746279 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:31.746244 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/rate-test-model-584ec5-kserve-6b8ddf6557-kfw2p"] Oct 30 02:45:33.739606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:33.739572 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" path="/var/lib/kubelet/pods/1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a/volumes" Oct 30 02:45:35.695548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:35.695511 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9"] Oct 30 02:45:35.739726 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:35.739695 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n"] Oct 30 02:45:35.870486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:35.870446 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerName="payload-processing" containerID="cri-o://d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4" gracePeriod=25 Oct 30 02:45:35.905249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:35.905185 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerName="payload-pre-processing" containerID="cri-o://570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a" gracePeriod=25 Oct 30 02:45:36.086507 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.086467 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" podUID="a1f64997-cd28-4d25-ad03-ddd332ac48fb" containerName="main" containerID="cri-o://6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3" gracePeriod=45 Oct 30 02:45:36.430431 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.430393 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:45:36.525629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.525594 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq\" (UniqueName: \"kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq\") pod \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " Oct 30 02:45:36.525837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.525645 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs\") pod \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " Oct 30 02:45:36.525837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.525684 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir\") pod \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " Oct 30 02:45:36.525837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.525725 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache\") pod \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " Oct 30 02:45:36.525837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.525774 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm\") pod \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " Oct 30 02:45:36.525837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.525808 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home\") pod \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\" (UID: \"a1f64997-cd28-4d25-ad03-ddd332ac48fb\") " Oct 30 02:45:36.526165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.526011 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir" pod "a1f64997-cd28-4d25-ad03-ddd332ac48fb" (UID: "a1f64997-cd28-4d25-ad03-ddd332ac48fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:36.526165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.526043 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache" pod "a1f64997-cd28-4d25-ad03-ddd332ac48fb" (UID: "a1f64997-cd28-4d25-ad03-ddd332ac48fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:36.526292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.526193 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home" pod "a1f64997-cd28-4d25-ad03-ddd332ac48fb" (UID: "a1f64997-cd28-4d25-ad03-ddd332ac48fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:36.526557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.526529 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:36.526616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.526558 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:36.526616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.526575 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:36.528054 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.528033 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm" pod "a1f64997-cd28-4d25-ad03-ddd332ac48fb" (UID: "a1f64997-cd28-4d25-ad03-ddd332ac48fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:36.528157 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.528104 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq" pod "a1f64997-cd28-4d25-ad03-ddd332ac48fb" (UID: "a1f64997-cd28-4d25-ad03-ddd332ac48fb"). InnerVolumeSpecName "kube-api-access-88dzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:36.528199 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.528154 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs" pod "a1f64997-cd28-4d25-ad03-ddd332ac48fb" (UID: "a1f64997-cd28-4d25-ad03-ddd332ac48fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:36.627732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.627696 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dzq\" (UniqueName: \"kubernetes.io/projected/a1f64997-cd28-4d25-ad03-ddd332ac48fb-kube-api-access-88dzq\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:36.627732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.627728 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1f64997-cd28-4d25-ad03-ddd332ac48fb-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:36.627732 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.627739 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1f64997-cd28-4d25-ad03-ddd332ac48fb-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:36.707419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.707327 6275 generic.go:356] "Generic (PLEG): container finished" podID="a1f64997-cd28-4d25-ad03-ddd332ac48fb" containerID="6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3" exitCode=0 Oct 30 02:45:36.707419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.707383 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" Oct 30 02:45:36.707419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.707410 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" event={"ID":"a1f64997-cd28-4d25-ad03-ddd332ac48fb","Type":"ContainerDied","Data":"6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3"} Oct 30 02:45:36.707998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.707452 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx" event={"ID":"a1f64997-cd28-4d25-ad03-ddd332ac48fb","Type":"ContainerDied","Data":"927eb73731cab94ba1c798bff8d31eed2d7c2dea60e8062f12ce90cd506904a4"} Oct 30 02:45:36.707998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.707475 6275 scope.go:122] "RemoveContainer" containerID="6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3" Oct 30 02:45:36.718086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.718065 6275 scope.go:122] "RemoveContainer" containerID="6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3" Oct 30 02:45:36.718544 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:36.718519 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3\": container with ID starting with 6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3 not found: ID does not exist" containerID="6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3" Oct 30 02:45:36.718702 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.718551 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3"} err="failed to get container status \"6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3\": rpc error: code = NotFound desc = could not find container \"6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3\": container with ID starting with 6d9c9a5488eb91a421a18737da5c008f6b73e38f688be37b119f809677d7a1d3 not found: ID does not exist" Oct 30 02:45:36.757705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.757672 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx"] Oct 30 02:45:36.768255 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:36.768224 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/rate-test-model-24fb97-kserve-67dcd9989-xpvtx"] Oct 30 02:45:37.740402 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:37.740370 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f64997-cd28-4d25-ad03-ddd332ac48fb" path="/var/lib/kubelet/pods/a1f64997-cd28-4d25-ad03-ddd332ac48fb/volumes" Oct 30 02:45:37.754884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:37.754851 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" podUID="bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" containerName="main" containerID="cri-o://7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3" gracePeriod=45 Oct 30 02:45:38.028219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.028195 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:45:38.043971 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.043939 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk\") pod \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " Oct 30 02:45:38.044164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.043980 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir\") pod \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " Oct 30 02:45:38.044164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044046 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home\") pod \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " Oct 30 02:45:38.044164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044161 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm\") pod \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " Oct 30 02:45:38.044372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044189 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs\") pod \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " Oct 30 02:45:38.044372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044256 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache\") pod \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\" (UID: \"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac\") " Oct 30 02:45:38.044372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044353 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir" pod "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" (UID: "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:38.044527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044360 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home" pod "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" (UID: "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:38.044585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044570 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache" pod "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" (UID: "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:38.044741 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044720 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:38.044808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044743 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:38.044808 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.044751 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:38.046326 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.046298 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm" pod "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" (UID: "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:38.046490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.046470 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk" pod "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" (UID: "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac"). InnerVolumeSpecName "kube-api-access-c95wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:38.046490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.046481 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs" pod "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" (UID: "bf70cf0e-c1ec-42dc-a4a7-244d89d835ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:38.147004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.146965 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-kube-api-access-c95wk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:38.147004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.147003 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:38.147263 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.147019 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:38.722733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.722700 6275 generic.go:356] "Generic (PLEG): container finished" podID="bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" containerID="7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3" exitCode=0 Oct 30 02:45:38.723035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.722746 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" event={"ID":"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac","Type":"ContainerDied","Data":"7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3"} Oct 30 02:45:38.723035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.722749 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" Oct 30 02:45:38.723035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.722772 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh" event={"ID":"bf70cf0e-c1ec-42dc-a4a7-244d89d835ac","Type":"ContainerDied","Data":"7f7de5fe700a3ea84a5ecb3d100bd2c9754cb76dc93bd8642d2f24d4b7a165f7"} Oct 30 02:45:38.723035 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.722793 6275 scope.go:122] "RemoveContainer" containerID="7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3" Oct 30 02:45:38.733521 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.733500 6275 scope.go:122] "RemoveContainer" containerID="7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3" Oct 30 02:45:38.733838 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:38.733811 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3\": container with ID starting with 7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3 not found: ID does not exist" containerID="7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3" Oct 30 02:45:38.733924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.733848 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3"} err="failed to get container status \"7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3\": rpc error: code = NotFound desc = could not find container \"7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3\": container with ID starting with 7ebec3c092c2972f85554412fdecf5b68d5e84ea1e85302f39abcbdae92b2aa3 not found: ID does not exist" Oct 30 02:45:38.772472 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.772433 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh"] Oct 30 02:45:38.776541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:38.776512 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/rate-test-model-01a9cd-kserve-6f4fc57c97-ggxqh"] Oct 30 02:45:39.738501 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:39.738467 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" path="/var/lib/kubelet/pods/bf70cf0e-c1ec-42dc-a4a7-244d89d835ac/volumes" Oct 30 02:45:40.695879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:40.695827 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" podUID="d8ccdb02-96b3-4b75-822c-e69ddfa45793" containerName="payload-processing" containerID="cri-o://553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8" gracePeriod=25 Oct 30 02:45:40.740412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:40.740372 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerName="payload-pre-processing" containerID="cri-o://ae555c9d5eb4bdd57a3f53b4463db00b0ac707db9fba2ec3dd887997eacd7839" gracePeriod=25 Oct 30 02:45:40.963978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:40.963917 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:45:41.149853 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.149821 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5\" (UniqueName: \"kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5\") pod \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " Oct 30 02:45:41.150043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.149895 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume\") pod \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\" (UID: \"d8ccdb02-96b3-4b75-822c-e69ddfa45793\") " Oct 30 02:45:41.150292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.150270 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume" pod "d8ccdb02-96b3-4b75-822c-e69ddfa45793" (UID: "d8ccdb02-96b3-4b75-822c-e69ddfa45793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:41.150549 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.150532 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ccdb02-96b3-4b75-822c-e69ddfa45793-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:41.152106 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.152082 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5" pod "d8ccdb02-96b3-4b75-822c-e69ddfa45793" (UID: "d8ccdb02-96b3-4b75-822c-e69ddfa45793"). InnerVolumeSpecName "kube-api-access-zpnc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:41.186934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.186899 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:45:41.251699 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.251622 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnc5\" (UniqueName: \"kubernetes.io/projected/d8ccdb02-96b3-4b75-822c-e69ddfa45793-kube-api-access-zpnc5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:41.728015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.727977 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:45:41.728511 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.728065 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:45:41.751983 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.751945 6275 generic.go:356] "Generic (PLEG): container finished" podID="d8ccdb02-96b3-4b75-822c-e69ddfa45793" containerID="553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8" exitCode=0 Oct 30 02:45:41.751983 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.751977 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" event={"ID":"d8ccdb02-96b3-4b75-822c-e69ddfa45793","Type":"ContainerDied","Data":"553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8"} Oct 30 02:45:41.752241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.751993 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" Oct 30 02:45:41.752241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.752027 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9" event={"ID":"d8ccdb02-96b3-4b75-822c-e69ddfa45793","Type":"ContainerDied","Data":"fbdbb99af9513ec1b76952587a064323963265d07eea5891cf247ab330dd329d"} Oct 30 02:45:41.752241 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.752048 6275 scope.go:122] "RemoveContainer" containerID="553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8" Oct 30 02:45:41.764022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.763994 6275 scope.go:122] "RemoveContainer" containerID="553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8" Oct 30 02:45:41.764310 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:41.764292 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8\": container with ID starting with 553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8 not found: ID does not exist" containerID="553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8" Oct 30 02:45:41.764378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.764318 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8"} err="failed to get container status \"553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8\": rpc error: code = NotFound desc = could not find container \"553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8\": container with ID starting with 553487415b29e993f7900f5950738c1ee32d5920af971d7631af63cf3c0f5ac8 not found: ID does not exist" Oct 30 02:45:41.799164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.799108 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9"] Oct 30 02:45:41.805086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:41.805054 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6b8bfa-6cbb86d6fd-mh9c9"] Oct 30 02:45:43.738984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:43.738942 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ccdb02-96b3-4b75-822c-e69ddfa45793" path="/var/lib/kubelet/pods/d8ccdb02-96b3-4b75-822c-e69ddfa45793/volumes" Oct 30 02:45:46.696555 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:46.696512 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:45:50.524231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:50.524195 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4"] Oct 30 02:45:50.524931 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:50.524899 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" containerID="cri-o://b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a" gracePeriod=30 Oct 30 02:45:51.185925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:51.185884 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:45:51.727343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:51.727304 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:45:51.727835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:51.727402 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:45:55.820617 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.820573 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:45:55.869803 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.869759 6275 generic.go:356] "Generic (PLEG): container finished" podID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerID="b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a" exitCode=0 Oct 30 02:45:55.869995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.869814 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" Oct 30 02:45:55.869995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.869840 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" event={"ID":"0c955d1c-d428-4ac7-b931-4cfe83e3ca92","Type":"ContainerDied","Data":"b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a"} Oct 30 02:45:55.869995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.869924 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4" event={"ID":"0c955d1c-d428-4ac7-b931-4cfe83e3ca92","Type":"ContainerDied","Data":"71162875e86c45fdbdfd684da08c575a552c7ae1e79037c58182a3562689621d"} Oct 30 02:45:55.869995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.869958 6275 scope.go:122] "RemoveContainer" containerID="b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a" Oct 30 02:45:55.882046 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.882018 6275 scope.go:122] "RemoveContainer" containerID="b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a" Oct 30 02:45:55.886559 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:55.886526 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a\": container with ID starting with b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a not found: ID does not exist" containerID="b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a" Oct 30 02:45:55.886705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.886563 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a"} err="failed to get container status \"b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a\": rpc error: code = NotFound desc = could not find container \"b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a\": container with ID starting with b9ad24aac15973718fb28d69cb57ef82e3e275deacdc4f6a495ff4e5f0652b2a not found: ID does not exist" Oct 30 02:45:55.961438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961406 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961468 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961520 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961557 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961760 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961590 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961760 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961646 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961760 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961717 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961760 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961746 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961797 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token\") pod \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\" (UID: \"0c955d1c-d428-4ac7-b931-4cfe83e3ca92\") " Oct 30 02:45:55.961987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961818 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:55.961987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961907 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:55.961987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.961923 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:55.962250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962039 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:55.962250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962174 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:55.962557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962531 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:55.962643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962564 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:55.962643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962578 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:55.962643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962592 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:55.962643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.962604 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:55.964403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.964379 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:45:55.964528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.964507 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "kube-api-access-bwjlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:55.964647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.964540 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:55.964814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:55.964794 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy" pod "0c955d1c-d428-4ac7-b931-4cfe83e3ca92" (UID: "0c955d1c-d428-4ac7-b931-4cfe83e3ca92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.065786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.063533 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.065786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.063563 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.065786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.063574 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-kube-api-access-bwjlt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.065786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.063584 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c955d1c-d428-4ac7-b931-4cfe83e3ca92-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.192525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.192496 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:45:56.208633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.208611 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:45:56.214794 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.214770 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4"] Oct 30 02:45:56.225222 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.225192 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9stc4"] Oct 30 02:45:56.367185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367099 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume\") pod \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " Oct 30 02:45:56.367348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367186 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs\" (UniqueName: \"kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs\") pod \"40d23e69-56ba-4550-89c9-d66cc6a895ed\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " Oct 30 02:45:56.367348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367235 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n\" (UniqueName: \"kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n\") pod \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\" (UID: \"01bdad21-9f15-43a1-8af7-1e22cf49aaa9\") " Oct 30 02:45:56.367348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367274 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume\") pod \"40d23e69-56ba-4550-89c9-d66cc6a895ed\" (UID: \"40d23e69-56ba-4550-89c9-d66cc6a895ed\") " Oct 30 02:45:56.367716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367690 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume" pod "40d23e69-56ba-4550-89c9-d66cc6a895ed" (UID: "40d23e69-56ba-4550-89c9-d66cc6a895ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:56.367716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367694 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume" pod "01bdad21-9f15-43a1-8af7-1e22cf49aaa9" (UID: "01bdad21-9f15-43a1-8af7-1e22cf49aaa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:56.367856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367780 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.367856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.367799 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d23e69-56ba-4550-89c9-d66cc6a895ed-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.369696 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.369670 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs" pod "40d23e69-56ba-4550-89c9-d66cc6a895ed" (UID: "40d23e69-56ba-4550-89c9-d66cc6a895ed"). InnerVolumeSpecName "kube-api-access-pxrcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.369810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.369695 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n" pod "01bdad21-9f15-43a1-8af7-1e22cf49aaa9" (UID: "01bdad21-9f15-43a1-8af7-1e22cf49aaa9"). InnerVolumeSpecName "kube-api-access-v7d8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.468751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.468710 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrcs\" (UniqueName: \"kubernetes.io/projected/40d23e69-56ba-4550-89c9-d66cc6a895ed-kube-api-access-pxrcs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.468751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.468741 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7d8n\" (UniqueName: \"kubernetes.io/projected/01bdad21-9f15-43a1-8af7-1e22cf49aaa9-kube-api-access-v7d8n\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.879003 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.878965 6275 generic.go:356] "Generic (PLEG): container finished" podID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerID="d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4" exitCode=0 Oct 30 02:45:56.879525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.879012 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" Oct 30 02:45:56.879525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.879046 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" event={"ID":"01bdad21-9f15-43a1-8af7-1e22cf49aaa9","Type":"ContainerDied","Data":"d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4"} Oct 30 02:45:56.879525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.879101 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2" event={"ID":"01bdad21-9f15-43a1-8af7-1e22cf49aaa9","Type":"ContainerDied","Data":"e910ebdb65ab1597f1a521d8b7c6e81bc5ef87c4a8d8c1835ec474863267584a"} Oct 30 02:45:56.879525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.879165 6275 scope.go:122] "RemoveContainer" containerID="d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4" Oct 30 02:45:56.880642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.880527 6275 generic.go:356] "Generic (PLEG): container finished" podID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerID="570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a" exitCode=0 Oct 30 02:45:56.880642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.880576 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" event={"ID":"40d23e69-56ba-4550-89c9-d66cc6a895ed","Type":"ContainerDied","Data":"570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a"} Oct 30 02:45:56.880642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.880587 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" Oct 30 02:45:56.880642 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.880616 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7" event={"ID":"40d23e69-56ba-4550-89c9-d66cc6a895ed","Type":"ContainerDied","Data":"811d62955a140225ddfa2ace9f99e384cef4dedfa89d046c521800fb4b1dca59"} Oct 30 02:45:56.891204 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.891182 6275 scope.go:122] "RemoveContainer" containerID="d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4" Oct 30 02:45:56.891554 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:56.891527 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4\": container with ID starting with d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4 not found: ID does not exist" containerID="d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4" Oct 30 02:45:56.891644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.891567 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4"} err="failed to get container status \"d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4\": rpc error: code = NotFound desc = could not find container \"d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4\": container with ID starting with d35787e5cf437f2f9dd50c3bbe3cef96bd7cfed386487d7ee9a11b9ef94bfbc4 not found: ID does not exist" Oct 30 02:45:56.891644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.891588 6275 scope.go:122] "RemoveContainer" containerID="570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a" Oct 30 02:45:56.900868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.900844 6275 scope.go:122] "RemoveContainer" containerID="570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a" Oct 30 02:45:56.901231 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:45:56.901209 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a\": container with ID starting with 570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a not found: ID does not exist" containerID="570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a" Oct 30 02:45:56.901297 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.901242 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a"} err="failed to get container status \"570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a\": rpc error: code = NotFound desc = could not find container \"570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a\": container with ID starting with 570f988eb09b517f6e28159512562ccc8ce4f55dd6667167348c34f3d9b6e67a not found: ID does not exist" Oct 30 02:45:56.928313 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.928281 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7"] Oct 30 02:45:56.930704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.930678 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-95f40f-5f5d6f56c-sjvh7"] Oct 30 02:45:56.961345 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.961315 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2"] Oct 30 02:45:56.967344 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:56.967317 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-95f40f-5ff9b95c8d-mjtg2"] Oct 30 02:45:57.739301 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:57.739265 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" path="/var/lib/kubelet/pods/01bdad21-9f15-43a1-8af7-1e22cf49aaa9/volumes" Oct 30 02:45:57.739992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:57.739971 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" path="/var/lib/kubelet/pods/0c955d1c-d428-4ac7-b931-4cfe83e3ca92/volumes" Oct 30 02:45:57.740578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:45:57.740557 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" path="/var/lib/kubelet/pods/40d23e69-56ba-4550-89c9-d66cc6a895ed/volumes" Oct 30 02:46:00.908789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:00.908755 6275 generic.go:356] "Generic (PLEG): container finished" podID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerID="ae555c9d5eb4bdd57a3f53b4463db00b0ac707db9fba2ec3dd887997eacd7839" exitCode=0 Oct 30 02:46:00.909231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:00.908832 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" event={"ID":"310a8418-7f15-4283-b7b7-ebf83096d94c","Type":"ContainerDied","Data":"ae555c9d5eb4bdd57a3f53b4463db00b0ac707db9fba2ec3dd887997eacd7839"} Oct 30 02:46:01.012313 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.012287 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:46:01.127553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.127513 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume\") pod \"310a8418-7f15-4283-b7b7-ebf83096d94c\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " Oct 30 02:46:01.127782 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.127763 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx\" (UniqueName: \"kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx\") pod \"310a8418-7f15-4283-b7b7-ebf83096d94c\" (UID: \"310a8418-7f15-4283-b7b7-ebf83096d94c\") " Oct 30 02:46:01.127922 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.127897 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume" pod "310a8418-7f15-4283-b7b7-ebf83096d94c" (UID: "310a8418-7f15-4283-b7b7-ebf83096d94c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:01.128280 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.128258 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310a8418-7f15-4283-b7b7-ebf83096d94c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:01.130087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.130060 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx" pod "310a8418-7f15-4283-b7b7-ebf83096d94c" (UID: "310a8418-7f15-4283-b7b7-ebf83096d94c"). InnerVolumeSpecName "kube-api-access-cz5wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:01.229376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.229342 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5wx\" (UniqueName: \"kubernetes.io/projected/310a8418-7f15-4283-b7b7-ebf83096d94c-kube-api-access-cz5wx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:01.916612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.916578 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" Oct 30 02:46:01.916612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.916598 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n" event={"ID":"310a8418-7f15-4283-b7b7-ebf83096d94c","Type":"ContainerDied","Data":"b9fc30c1c4551deeeaad69639fc47ca75ef00f551a8a30d3f006a4a849c5bc39"} Oct 30 02:46:01.917068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.916699 6275 scope.go:122] "RemoveContainer" containerID="ae555c9d5eb4bdd57a3f53b4463db00b0ac707db9fba2ec3dd887997eacd7839" Oct 30 02:46:01.962894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.962853 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n"] Oct 30 02:46:01.973152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:01.973103 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6b8bfa-6d5656f495-5sr8n"] Oct 30 02:46:02.571609 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:02.571568 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk"] Oct 30 02:46:02.609441 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:02.609404 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf"] Oct 30 02:46:03.739537 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:03.739506 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" path="/var/lib/kubelet/pods/310a8418-7f15-4283-b7b7-ebf83096d94c/volumes" Oct 30 02:46:07.572228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:07.572180 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" containerName="payload-processing" containerID="cri-o://13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363" gracePeriod=25 Oct 30 02:46:07.610650 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:07.610585 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" containerName="payload-pre-processing" containerID="cri-o://6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6" gracePeriod=25 Oct 30 02:46:12.165170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:12.165068 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462"] Oct 30 02:46:12.829474 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:12.829429 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:12.830013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:12.829973 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:16.699280 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:16.699240 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:46:17.165605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:17.165554 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerName="payload-processing" containerID="cri-o://e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577" gracePeriod=25 Oct 30 02:46:19.686454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:19.686418 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw"] Oct 30 02:46:19.686899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:19.686640 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" podUID="37349c06-fce3-45a9-9128-344fa1508a06" containerName="maas-api" containerID="cri-o://d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8" gracePeriod=30 Oct 30 02:46:19.989992 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:19.989960 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:46:20.092906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.091261 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs\") pod \"37349c06-fce3-45a9-9128-344fa1508a06\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " Oct 30 02:46:20.092906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.091324 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls\") pod \"37349c06-fce3-45a9-9128-344fa1508a06\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " Oct 30 02:46:20.092906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.091437 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm\" (UniqueName: \"kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm\") pod \"37349c06-fce3-45a9-9128-344fa1508a06\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " Oct 30 02:46:20.092906 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.091503 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca\") pod \"37349c06-fce3-45a9-9128-344fa1508a06\" (UID: \"37349c06-fce3-45a9-9128-344fa1508a06\") " Oct 30 02:46:20.094979 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.094945 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca" pod "37349c06-fce3-45a9-9128-344fa1508a06" (UID: "37349c06-fce3-45a9-9128-344fa1508a06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:20.099413 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.099388 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs" pod "37349c06-fce3-45a9-9128-344fa1508a06" (UID: "37349c06-fce3-45a9-9128-344fa1508a06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:20.109698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.102857 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm" pod "37349c06-fce3-45a9-9128-344fa1508a06" (UID: "37349c06-fce3-45a9-9128-344fa1508a06"). InnerVolumeSpecName "kube-api-access-gjtdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:20.109698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.105276 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls" pod "37349c06-fce3-45a9-9128-344fa1508a06" (UID: "37349c06-fce3-45a9-9128-344fa1508a06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:20.120674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.120629 6275 generic.go:356] "Generic (PLEG): container finished" podID="37349c06-fce3-45a9-9128-344fa1508a06" containerID="d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8" exitCode=0 Oct 30 02:46:20.120820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.120778 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" event={"ID":"37349c06-fce3-45a9-9128-344fa1508a06","Type":"ContainerDied","Data":"d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8"} Oct 30 02:46:20.120881 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.120865 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" event={"ID":"37349c06-fce3-45a9-9128-344fa1508a06","Type":"ContainerDied","Data":"c5907da282d11d24a5fdbd615942d8352eb5da3b5151a5b1d37dbc80b87db806"} Oct 30 02:46:20.120921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.120896 6275 scope.go:122] "RemoveContainer" containerID="d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8" Oct 30 02:46:20.121061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.121048 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw" Oct 30 02:46:20.137152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.137115 6275 scope.go:122] "RemoveContainer" containerID="d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8" Oct 30 02:46:20.137494 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:20.137469 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8\": container with ID starting with d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8 not found: ID does not exist" containerID="d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8" Oct 30 02:46:20.137589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.137503 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8"} err="failed to get container status \"d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8\": rpc error: code = NotFound desc = could not find container \"d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8\": container with ID starting with d41de63704fd1f9e6fd6c3abbf83d1d123ca1cf9fc85887b58ca85683e4cd3b8 not found: ID does not exist" Oct 30 02:46:20.178099 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.178063 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw"] Oct 30 02:46:20.181766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.181734 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0fc73-7769f7d65c-h8ltw"] Oct 30 02:46:20.193761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.193732 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:20.193761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.193756 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37349c06-fce3-45a9-9128-344fa1508a06-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:20.193761 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.193767 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjtdm\" (UniqueName: \"kubernetes.io/projected/37349c06-fce3-45a9-9128-344fa1508a06-kube-api-access-gjtdm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:20.193950 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:20.193776 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37349c06-fce3-45a9-9128-344fa1508a06-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:21.738820 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:21.738784 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37349c06-fce3-45a9-9128-344fa1508a06" path="/var/lib/kubelet/pods/37349c06-fce3-45a9-9128-344fa1508a06/volumes" Oct 30 02:46:22.821557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:22.821519 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:22.821955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:22.821565 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:23.859954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:23.859915 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:25.005710 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.005673 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7"] Oct 30 02:46:25.006231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.005938 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" podUID="236d9bfe-9a3f-4240-80a6-c01b604d31b0" containerName="maas-api" containerID="cri-o://0436b5b6e40825b0dbd10e0f3b7e4295ea9840bdddaafc02559985fcd89de4b8" gracePeriod=30 Oct 30 02:46:25.162811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.162779 6275 generic.go:356] "Generic (PLEG): container finished" podID="236d9bfe-9a3f-4240-80a6-c01b604d31b0" containerID="0436b5b6e40825b0dbd10e0f3b7e4295ea9840bdddaafc02559985fcd89de4b8" exitCode=0 Oct 30 02:46:25.162978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.162843 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" event={"ID":"236d9bfe-9a3f-4240-80a6-c01b604d31b0","Type":"ContainerDied","Data":"0436b5b6e40825b0dbd10e0f3b7e4295ea9840bdddaafc02559985fcd89de4b8"} Oct 30 02:46:25.298259 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.298223 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:46:25.468033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.459936 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs\") pod \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " Oct 30 02:46:25.468033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.460014 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca\") pod \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " Oct 30 02:46:25.468033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.460067 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt\" (UniqueName: \"kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt\") pod \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " Oct 30 02:46:25.468033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.460242 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls\") pod \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\" (UID: \"236d9bfe-9a3f-4240-80a6-c01b604d31b0\") " Oct 30 02:46:25.479023 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.473528 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca" pod "236d9bfe-9a3f-4240-80a6-c01b604d31b0" (UID: "236d9bfe-9a3f-4240-80a6-c01b604d31b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:25.491921 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.491880 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs" pod "236d9bfe-9a3f-4240-80a6-c01b604d31b0" (UID: "236d9bfe-9a3f-4240-80a6-c01b604d31b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:25.492691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.492656 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt" pod "236d9bfe-9a3f-4240-80a6-c01b604d31b0" (UID: "236d9bfe-9a3f-4240-80a6-c01b604d31b0"). InnerVolumeSpecName "kube-api-access-wv2bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:25.492862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.492691 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls" pod "236d9bfe-9a3f-4240-80a6-c01b604d31b0" (UID: "236d9bfe-9a3f-4240-80a6-c01b604d31b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:25.562216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.562115 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:25.562216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.562169 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236d9bfe-9a3f-4240-80a6-c01b604d31b0-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:25.562216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.562184 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236d9bfe-9a3f-4240-80a6-c01b604d31b0-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:25.562216 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:25.562198 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2bt\" (UniqueName: \"kubernetes.io/projected/236d9bfe-9a3f-4240-80a6-c01b604d31b0-kube-api-access-wv2bt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:26.170476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:26.170447 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" Oct 30 02:46:26.170476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:26.170452 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7" event={"ID":"236d9bfe-9a3f-4240-80a6-c01b604d31b0","Type":"ContainerDied","Data":"c8b6c1b9a079b2cf1584098f230dfa020f8aea2bf1455c8d19a039b8f8598d56"} Oct 30 02:46:26.170991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:26.170514 6275 scope.go:122] "RemoveContainer" containerID="0436b5b6e40825b0dbd10e0f3b7e4295ea9840bdddaafc02559985fcd89de4b8" Oct 30 02:46:26.217798 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:26.217763 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7"] Oct 30 02:46:26.227903 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:26.227864 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9dfb3c-bd75b5789-66ds7"] Oct 30 02:46:27.743016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.742981 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236d9bfe-9a3f-4240-80a6-c01b604d31b0" path="/var/lib/kubelet/pods/236d9bfe-9a3f-4240-80a6-c01b604d31b0/volumes" Oct 30 02:46:27.880236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.880207 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:46:27.924847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.924818 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:46:27.993172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.993106 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume\") pod \"7413499b-ba8b-4489-bc32-2655b26ab793\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " Oct 30 02:46:27.993344 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.993193 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v\" (UniqueName: \"kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v\") pod \"7413499b-ba8b-4489-bc32-2655b26ab793\" (UID: \"7413499b-ba8b-4489-bc32-2655b26ab793\") " Oct 30 02:46:27.993612 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.993588 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume" pod "7413499b-ba8b-4489-bc32-2655b26ab793" (UID: "7413499b-ba8b-4489-bc32-2655b26ab793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:27.995673 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:27.995649 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v" pod "7413499b-ba8b-4489-bc32-2655b26ab793" (UID: "7413499b-ba8b-4489-bc32-2655b26ab793"). InnerVolumeSpecName "kube-api-access-jc89v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:28.094226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.094144 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume\") pod \"369285b9-8086-43a0-a3df-36de41f1ea34\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " Oct 30 02:46:28.094383 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.094279 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d\" (UniqueName: \"kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d\") pod \"369285b9-8086-43a0-a3df-36de41f1ea34\" (UID: \"369285b9-8086-43a0-a3df-36de41f1ea34\") " Oct 30 02:46:28.094530 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.094508 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume" pod "369285b9-8086-43a0-a3df-36de41f1ea34" (UID: "369285b9-8086-43a0-a3df-36de41f1ea34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:28.094746 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.094727 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369285b9-8086-43a0-a3df-36de41f1ea34-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:28.094818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.094748 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7413499b-ba8b-4489-bc32-2655b26ab793-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:28.094818 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.094759 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc89v\" (UniqueName: \"kubernetes.io/projected/7413499b-ba8b-4489-bc32-2655b26ab793-kube-api-access-jc89v\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:28.096494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.096471 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d" pod "369285b9-8086-43a0-a3df-36de41f1ea34" (UID: "369285b9-8086-43a0-a3df-36de41f1ea34"). InnerVolumeSpecName "kube-api-access-hls7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:28.184669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.184636 6275 generic.go:356] "Generic (PLEG): container finished" podID="369285b9-8086-43a0-a3df-36de41f1ea34" containerID="6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6" exitCode=0 Oct 30 02:46:28.184839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.184692 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" Oct 30 02:46:28.184839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.184717 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" event={"ID":"369285b9-8086-43a0-a3df-36de41f1ea34","Type":"ContainerDied","Data":"6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6"} Oct 30 02:46:28.184839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.184778 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf" event={"ID":"369285b9-8086-43a0-a3df-36de41f1ea34","Type":"ContainerDied","Data":"a5123cdcf0719785ab6c19626d6bb5b7c171ae7f54391ce0f29eda8da03d6b16"} Oct 30 02:46:28.184839 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.184808 6275 scope.go:122] "RemoveContainer" containerID="6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6" Oct 30 02:46:28.186516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.186325 6275 generic.go:356] "Generic (PLEG): container finished" podID="7413499b-ba8b-4489-bc32-2655b26ab793" containerID="13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363" exitCode=0 Oct 30 02:46:28.186516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.186385 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" Oct 30 02:46:28.186516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.186437 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" event={"ID":"7413499b-ba8b-4489-bc32-2655b26ab793","Type":"ContainerDied","Data":"13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363"} Oct 30 02:46:28.186516 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.186479 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk" event={"ID":"7413499b-ba8b-4489-bc32-2655b26ab793","Type":"ContainerDied","Data":"88741ae5ee663d960d6d5a9f9065067f06293f81a9b9bc3ef71475ca2a783626"} Oct 30 02:46:28.196232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.196209 6275 scope.go:122] "RemoveContainer" containerID="6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6" Oct 30 02:46:28.196333 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.196278 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hls7d\" (UniqueName: \"kubernetes.io/projected/369285b9-8086-43a0-a3df-36de41f1ea34-kube-api-access-hls7d\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:28.196662 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:28.196514 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6\": container with ID starting with 6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6 not found: ID does not exist" containerID="6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6" Oct 30 02:46:28.196662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.196547 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6"} err="failed to get container status \"6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6\": rpc error: code = NotFound desc = could not find container \"6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6\": container with ID starting with 6fd346aa75dd8e4b281ba116f2470bce25d0a3cd2bd917cff56057e7bec086e6 not found: ID does not exist" Oct 30 02:46:28.196662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.196570 6275 scope.go:122] "RemoveContainer" containerID="13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363" Oct 30 02:46:28.205420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.205400 6275 scope.go:122] "RemoveContainer" containerID="13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363" Oct 30 02:46:28.205690 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:28.205667 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363\": container with ID starting with 13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363 not found: ID does not exist" containerID="13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363" Oct 30 02:46:28.205768 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.205696 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363"} err="failed to get container status \"13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363\": rpc error: code = NotFound desc = could not find container \"13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363\": container with ID starting with 13b2eb883a9a81e6c53804ea84c0711064b6484b6d20172efe6ae98851b4c363 not found: ID does not exist" Oct 30 02:46:28.244142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.241171 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf"] Oct 30 02:46:28.247838 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.244944 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0fc73-7b84b6d8ff-ccvzf"] Oct 30 02:46:28.270059 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.270027 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk"] Oct 30 02:46:28.273586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.273561 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0fc73-5b8bddf64d-v62bk"] Oct 30 02:46:28.995255 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.995150 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp"] Oct 30 02:46:28.995902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:28.995363 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" containerID="cri-o://bcc7e7884a17834a910be4a279d396457c4ce6ed8a3b2b6229848380c1ce0613" gracePeriod=30 Oct 30 02:46:29.740446 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:29.740413 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" path="/var/lib/kubelet/pods/369285b9-8086-43a0-a3df-36de41f1ea34/volumes" Oct 30 02:46:29.740851 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:29.740837 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" path="/var/lib/kubelet/pods/7413499b-ba8b-4489-bc32-2655b26ab793/volumes" Oct 30 02:46:33.859116 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:33.859076 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:34.227913 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.227866 6275 generic.go:356] "Generic (PLEG): container finished" podID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerID="bcc7e7884a17834a910be4a279d396457c4ce6ed8a3b2b6229848380c1ce0613" exitCode=0 Oct 30 02:46:34.228171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.227933 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" event={"ID":"5c06325f-3df6-48c3-8f02-53aeef8a2de2","Type":"ContainerDied","Data":"bcc7e7884a17834a910be4a279d396457c4ce6ed8a3b2b6229848380c1ce0613"} Oct 30 02:46:34.228171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.228001 6275 scope.go:122] "RemoveContainer" containerID="49c562bea926f8a407dc2a17b91ebf66400f7d0259fee0ea4dd2fa8074a08dfa" Oct 30 02:46:34.276787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.276763 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:46:34.474868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.474783 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.474868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.474826 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.474868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.474858 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.474883 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.474967 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.474999 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475038 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475169 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:34.475471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475204 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475250 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo\") pod \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\" (UID: \"5c06325f-3df6-48c3-8f02-53aeef8a2de2\") " Oct 30 02:46:34.475471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475320 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:34.475471 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475331 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:34.475707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475538 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:34.475707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475622 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:34.475786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475715 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.475786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475737 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.475786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475750 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.475786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475763 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.475786 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.475776 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.477368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.477334 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:34.477368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.477342 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:34.477580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.477560 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:34.477620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.477564 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9" pod "5c06325f-3df6-48c3-8f02-53aeef8a2de2" (UID: "5c06325f-3df6-48c3-8f02-53aeef8a2de2"). InnerVolumeSpecName "kube-api-access-5nlf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:34.577088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.577048 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.577088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.577081 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlf9\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-kube-api-access-5nlf9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.577088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.577094 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.577363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:34.577106 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c06325f-3df6-48c3-8f02-53aeef8a2de2-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:35.235716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:35.235684 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" Oct 30 02:46:35.236220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:35.235712 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp" event={"ID":"5c06325f-3df6-48c3-8f02-53aeef8a2de2","Type":"ContainerDied","Data":"c2e7c5e035bf90ad7148fea802ca8cbd0a07e0ca71cc41821f45a566bd0e298e"} Oct 30 02:46:35.236220 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:35.235793 6275 scope.go:122] "RemoveContainer" containerID="bcc7e7884a17834a910be4a279d396457c4ce6ed8a3b2b6229848380c1ce0613" Oct 30 02:46:35.295558 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:35.295519 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp"] Oct 30 02:46:35.305465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:35.305442 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-nhlpp"] Oct 30 02:46:35.739669 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:35.739635 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" path="/var/lib/kubelet/pods/5c06325f-3df6-48c3-8f02-53aeef8a2de2/volumes" Oct 30 02:46:36.323056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:36.323023 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k"] Oct 30 02:46:36.357474 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:36.357401 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz"] Oct 30 02:46:37.427997 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.427929 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:46:37.510328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.510290 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r\") pod \"6d6fe799-f177-4de5-b3d1-be217af32a1e\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " Oct 30 02:46:37.510508 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.510411 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume\") pod \"6d6fe799-f177-4de5-b3d1-be217af32a1e\" (UID: \"6d6fe799-f177-4de5-b3d1-be217af32a1e\") " Oct 30 02:46:37.510862 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.510840 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume" pod "6d6fe799-f177-4de5-b3d1-be217af32a1e" (UID: "6d6fe799-f177-4de5-b3d1-be217af32a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:37.512717 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.512682 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r" pod "6d6fe799-f177-4de5-b3d1-be217af32a1e" (UID: "6d6fe799-f177-4de5-b3d1-be217af32a1e"). InnerVolumeSpecName "kube-api-access-mhs6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:37.611966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.611929 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6fe799-f177-4de5-b3d1-be217af32a1e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:37.611966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:37.611965 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhs6r\" (UniqueName: \"kubernetes.io/projected/6d6fe799-f177-4de5-b3d1-be217af32a1e-kube-api-access-mhs6r\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:38.266707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.266658 6275 generic.go:356] "Generic (PLEG): container finished" podID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerID="e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577" exitCode=0 Oct 30 02:46:38.266900 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.266869 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" event={"ID":"6d6fe799-f177-4de5-b3d1-be217af32a1e","Type":"ContainerDied","Data":"e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577"} Oct 30 02:46:38.266961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.266923 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" event={"ID":"6d6fe799-f177-4de5-b3d1-be217af32a1e","Type":"ContainerDied","Data":"72ed956221037e69f10277d8dbdafbabdd17a09de889db13acdc0bb2a81d5732"} Oct 30 02:46:38.266961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.266953 6275 scope.go:122] "RemoveContainer" containerID="e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577" Oct 30 02:46:38.267088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.267075 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462" Oct 30 02:46:38.280860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.280698 6275 scope.go:122] "RemoveContainer" containerID="e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577" Oct 30 02:46:38.281154 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:38.281100 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577\": container with ID starting with e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577 not found: ID does not exist" containerID="e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577" Oct 30 02:46:38.281260 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.281168 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577"} err="failed to get container status \"e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577\": rpc error: code = NotFound desc = could not find container \"e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577\": container with ID starting with e9000338bbfbc9bc5108f1a80502fa5e6bcda6bd04e27f44a4d6fb17a565a577 not found: ID does not exist" Oct 30 02:46:38.322014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.321977 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462"] Oct 30 02:46:38.330207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:38.330179 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9dfb3c-684d7bc7cb-pl462"] Oct 30 02:46:39.738242 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:39.738202 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" path="/var/lib/kubelet/pods/6d6fe799-f177-4de5-b3d1-be217af32a1e/volumes" Oct 30 02:46:41.324412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:41.324359 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerName="payload-processing" containerID="cri-o://b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108" gracePeriod=25 Oct 30 02:46:41.358382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:41.358345 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerName="payload-pre-processing" containerID="cri-o://734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90" gracePeriod=25 Oct 30 02:46:44.992807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:44.992770 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng"] Oct 30 02:46:45.025520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:45.025487 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj"] Oct 30 02:46:46.718757 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:46.718720 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:46:47.911927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:47.911068 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r"] Oct 30 02:46:47.911927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:47.911340 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" podUID="4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" containerName="maas-api" containerID="cri-o://bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26" gracePeriod=30 Oct 30 02:46:48.202594 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.202566 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:46:48.259756 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.259723 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs\") pod \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " Oct 30 02:46:48.259961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.259828 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls\") pod \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " Oct 30 02:46:48.259961 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.259957 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca\") pod \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " Oct 30 02:46:48.260094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.259980 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6\" (UniqueName: \"kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6\") pod \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\" (UID: \"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab\") " Oct 30 02:46:48.260559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.260347 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca" pod "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" (UID: "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:48.260559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.260494 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:48.266436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.263322 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls" pod "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" (UID: "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:48.266436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.263601 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6" pod "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" (UID: "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab"). InnerVolumeSpecName "kube-api-access-pl4s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:48.266436 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.264479 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs" pod "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" (UID: "4eb67639-18bc-4b31-8ff8-dde4cdeb81ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:48.330248 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.330216 6275 generic.go:356] "Generic (PLEG): container finished" podID="4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" containerID="bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26" exitCode=0 Oct 30 02:46:48.330426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.330264 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" Oct 30 02:46:48.330426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.330301 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" event={"ID":"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab","Type":"ContainerDied","Data":"bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26"} Oct 30 02:46:48.330426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.330355 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r" event={"ID":"4eb67639-18bc-4b31-8ff8-dde4cdeb81ab","Type":"ContainerDied","Data":"d1a8dd71aa7cee9a9b435ec1b559a3570c3a6e2e67c9bde312cfbbc96bdbcc08"} Oct 30 02:46:48.330426 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.330377 6275 scope.go:122] "RemoveContainer" containerID="bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26" Oct 30 02:46:48.341086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.341059 6275 scope.go:122] "RemoveContainer" containerID="bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26" Oct 30 02:46:48.341476 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:46:48.341445 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26\": container with ID starting with bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26 not found: ID does not exist" containerID="bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26" Oct 30 02:46:48.341568 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.341489 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26"} err="failed to get container status \"bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26\": rpc error: code = NotFound desc = could not find container \"bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26\": container with ID starting with bc5ef5dc04bac4b63e692e3d2cc451b56e86eea781d86ccf12c72cb4e277ec26 not found: ID does not exist" Oct 30 02:46:48.378145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.372404 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4s6\" (UniqueName: \"kubernetes.io/projected/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-kube-api-access-pl4s6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:48.378145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.372445 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:48.378145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.372462 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:46:48.381164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.380581 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r"] Oct 30 02:46:48.384215 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:48.384185 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-01a9cd-769c789d98-b4n4r"] Oct 30 02:46:49.551857 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:49.551817 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:49.552278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:49.551856 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:49.739886 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:49.739839 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" path="/var/lib/kubelet/pods/4eb67639-18bc-4b31-8ff8-dde4cdeb81ab/volumes" Oct 30 02:46:49.993325 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:49.993241 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerName="payload-processing" containerID="cri-o://92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3" gracePeriod=25 Oct 30 02:46:50.025964 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:50.025919 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" containerName="payload-pre-processing" containerID="cri-o://25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac" gracePeriod=25 Oct 30 02:46:54.164531 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:54.163915 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm"] Oct 30 02:46:54.197942 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:54.197909 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc"] Oct 30 02:46:57.108935 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:57.108884 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:57.109432 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:57.108931 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:59.164292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:59.164249 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerName="payload-processing" containerID="cri-o://82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201" gracePeriod=25 Oct 30 02:46:59.198883 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:59.198842 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerName="payload-pre-processing" containerID="cri-o://139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88" gracePeriod=25 Oct 30 02:46:59.525033 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:59.524997 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:59.552229 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:59.552198 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:59.552379 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:46:59.552203 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:00.260700 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.260656 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m"] Oct 30 02:47:00.261247 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.260912 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" podUID="c564e47b-d931-4e9f-a58e-00d3f9d314a9" containerName="maas-api" containerID="cri-o://19f5399a46a7d5663e2578648ec76d2c278de684200fa41ea2703349dd679125" gracePeriod=30 Oct 30 02:47:00.402623 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.402589 6275 generic.go:356] "Generic (PLEG): container finished" podID="c564e47b-d931-4e9f-a58e-00d3f9d314a9" containerID="19f5399a46a7d5663e2578648ec76d2c278de684200fa41ea2703349dd679125" exitCode=0 Oct 30 02:47:00.402793 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.402658 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" event={"ID":"c564e47b-d931-4e9f-a58e-00d3f9d314a9","Type":"ContainerDied","Data":"19f5399a46a7d5663e2578648ec76d2c278de684200fa41ea2703349dd679125"} Oct 30 02:47:00.449686 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.449656 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt"] Oct 30 02:47:00.519977 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.519913 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:47:00.532290 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.532257 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca\") pod \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " Oct 30 02:47:00.532462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.532324 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs\") pod \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " Oct 30 02:47:00.532462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.532380 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk\" (UniqueName: \"kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk\") pod \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " Oct 30 02:47:00.532462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.532443 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls\") pod \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\" (UID: \"c564e47b-d931-4e9f-a58e-00d3f9d314a9\") " Oct 30 02:47:00.532811 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.532781 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca" pod "c564e47b-d931-4e9f-a58e-00d3f9d314a9" (UID: "c564e47b-d931-4e9f-a58e-00d3f9d314a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:00.534691 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.534665 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs" pod "c564e47b-d931-4e9f-a58e-00d3f9d314a9" (UID: "c564e47b-d931-4e9f-a58e-00d3f9d314a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:00.534788 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.534687 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls" pod "c564e47b-d931-4e9f-a58e-00d3f9d314a9" (UID: "c564e47b-d931-4e9f-a58e-00d3f9d314a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:00.535016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.535002 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk" pod "c564e47b-d931-4e9f-a58e-00d3f9d314a9" (UID: "c564e47b-d931-4e9f-a58e-00d3f9d314a9"). InnerVolumeSpecName "kube-api-access-sd9vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:00.633804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.633767 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.633804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.633798 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c564e47b-d931-4e9f-a58e-00d3f9d314a9-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.633804 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.633808 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c564e47b-d931-4e9f-a58e-00d3f9d314a9-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.634055 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:00.633817 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9vk\" (UniqueName: \"kubernetes.io/projected/c564e47b-d931-4e9f-a58e-00d3f9d314a9-kube-api-access-sd9vk\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:01.409200 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.409161 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" event={"ID":"c564e47b-d931-4e9f-a58e-00d3f9d314a9","Type":"ContainerDied","Data":"d4abc5b05b3fc914c3c304f50d831eb4bba924e6dca48dba7f14b53067de34e6"} Oct 30 02:47:01.409597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.409174 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m" Oct 30 02:47:01.409597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.409271 6275 scope.go:122] "RemoveContainer" containerID="19f5399a46a7d5663e2578648ec76d2c278de684200fa41ea2703349dd679125" Oct 30 02:47:01.479569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.479542 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m"] Oct 30 02:47:01.489787 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.489758 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2819d2-d7cdcddb5-skh8m"] Oct 30 02:47:01.624380 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.624348 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:47:01.644723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.644697 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm\" (UniqueName: \"kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm\") pod \"e530cbf4-a38c-4a03-a242-948d62ece7d7\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " Oct 30 02:47:01.644844 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.644759 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume\") pod \"e530cbf4-a38c-4a03-a242-948d62ece7d7\" (UID: \"e530cbf4-a38c-4a03-a242-948d62ece7d7\") " Oct 30 02:47:01.645190 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.645164 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume" pod "e530cbf4-a38c-4a03-a242-948d62ece7d7" (UID: "e530cbf4-a38c-4a03-a242-948d62ece7d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:01.645648 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.645623 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:47:01.646704 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.646684 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm" pod "e530cbf4-a38c-4a03-a242-948d62ece7d7" (UID: "e530cbf4-a38c-4a03-a242-948d62ece7d7"). InnerVolumeSpecName "kube-api-access-d2mjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:01.739232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.739197 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c564e47b-d931-4e9f-a58e-00d3f9d314a9" path="/var/lib/kubelet/pods/c564e47b-d931-4e9f-a58e-00d3f9d314a9/volumes" Oct 30 02:47:01.745430 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.745403 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume\") pod \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " Oct 30 02:47:01.745588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.745453 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw\" (UniqueName: \"kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw\") pod \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\" (UID: \"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e\") " Oct 30 02:47:01.745755 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.745734 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume" pod "67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" (UID: "67ce7a1c-cc4e-4576-b408-a3ea63a88c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:01.745864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.745743 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mjm\" (UniqueName: \"kubernetes.io/projected/e530cbf4-a38c-4a03-a242-948d62ece7d7-kube-api-access-d2mjm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:01.745864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.745800 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e530cbf4-a38c-4a03-a242-948d62ece7d7-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:01.747645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.747624 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw" pod "67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" (UID: "67ce7a1c-cc4e-4576-b408-a3ea63a88c1e"). InnerVolumeSpecName "kube-api-access-7bkjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:01.846579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.846495 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:01.846579 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:01.846527 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bkjw\" (UniqueName: \"kubernetes.io/projected/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e-kube-api-access-7bkjw\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.416708 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.416674 6275 generic.go:356] "Generic (PLEG): container finished" podID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerID="b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108" exitCode=0 Oct 30 02:47:02.417185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.416718 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" Oct 30 02:47:02.417185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.416747 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" event={"ID":"e530cbf4-a38c-4a03-a242-948d62ece7d7","Type":"ContainerDied","Data":"b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108"} Oct 30 02:47:02.417185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.416801 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k" event={"ID":"e530cbf4-a38c-4a03-a242-948d62ece7d7","Type":"ContainerDied","Data":"08d03aaf658765a57cc152996b5d5c46b293c3fc7252d524a9de0012bf55f043"} Oct 30 02:47:02.417185 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.416827 6275 scope.go:122] "RemoveContainer" containerID="b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108" Oct 30 02:47:02.418338 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.418313 6275 generic.go:356] "Generic (PLEG): container finished" podID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerID="734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90" exitCode=0 Oct 30 02:47:02.418478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.418345 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" event={"ID":"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e","Type":"ContainerDied","Data":"734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90"} Oct 30 02:47:02.418478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.418366 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" Oct 30 02:47:02.418478 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.418395 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz" event={"ID":"67ce7a1c-cc4e-4576-b408-a3ea63a88c1e","Type":"ContainerDied","Data":"7fae047eb939ced6091dd9c462015effeaf7425f7359b16d21c22e929b9f80f6"} Oct 30 02:47:02.426422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.426370 6275 scope.go:122] "RemoveContainer" containerID="b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108" Oct 30 02:47:02.426709 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:02.426647 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108\": container with ID starting with b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108 not found: ID does not exist" containerID="b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108" Oct 30 02:47:02.426709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.426690 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108"} err="failed to get container status \"b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108\": rpc error: code = NotFound desc = could not find container \"b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108\": container with ID starting with b149a149a6b5e30b76c8e3fbea0a40b78b91e834ce7096ef6b2b9f166af00108 not found: ID does not exist" Oct 30 02:47:02.426709 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.426709 6275 scope.go:122] "RemoveContainer" containerID="734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90" Oct 30 02:47:02.434841 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.434820 6275 scope.go:122] "RemoveContainer" containerID="734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90" Oct 30 02:47:02.435097 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:02.435078 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90\": container with ID starting with 734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90 not found: ID does not exist" containerID="734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90" Oct 30 02:47:02.435192 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.435103 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90"} err="failed to get container status \"734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90\": rpc error: code = NotFound desc = could not find container \"734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90\": container with ID starting with 734d1f364e6f2b715523a8a3939bc9f2d16bf1c9c554a39c3b7fa383d99a9e90 not found: ID does not exist" Oct 30 02:47:02.463625 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.463590 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k"] Oct 30 02:47:02.474131 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.474094 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0fdc4e-6fb5b8d96b-lvw2k"] Oct 30 02:47:02.499985 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.499954 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz"] Oct 30 02:47:02.510698 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:02.510664 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0fdc4e-dbc9d6c8c-7fbgz"] Oct 30 02:47:03.216231 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:03.216197 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg"] Oct 30 02:47:03.216490 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:03.216462 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" containerID="cri-o://6392b21109e286456f3f615a49c9c47bacba116912560c5c8bcea81ea9940e43" gracePeriod=30 Oct 30 02:47:03.739445 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:03.739413 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" path="/var/lib/kubelet/pods/67ce7a1c-cc4e-4576-b408-a3ea63a88c1e/volumes" Oct 30 02:47:03.739866 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:03.739850 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" path="/var/lib/kubelet/pods/e530cbf4-a38c-4a03-a242-948d62ece7d7/volumes" Oct 30 02:47:06.226207 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:06.226164 6275 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:47:06.226598 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:06.226228 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:47:07.108917 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:07.108878 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:07.109114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:07.108882 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:08.459043 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.459003 6275 generic.go:356] "Generic (PLEG): container finished" podID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerID="6392b21109e286456f3f615a49c9c47bacba116912560c5c8bcea81ea9940e43" exitCode=0 Oct 30 02:47:08.459529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.459077 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" event={"ID":"f789454e-6ef6-47d1-b92d-b18248f1ab9b","Type":"ContainerDied","Data":"6392b21109e286456f3f615a49c9c47bacba116912560c5c8bcea81ea9940e43"} Oct 30 02:47:08.459529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.459207 6275 scope.go:122] "RemoveContainer" containerID="0b0ae3f2a23417584746ac29b577d2786519fa523b6a3bf578e135c65b545a88" Oct 30 02:47:08.510675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.510650 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:47:08.515465 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.515430 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:08.615392 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615361 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615442 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615502 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615569 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615544 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615574 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615628 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615671 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:08.615928 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615690 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.615987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615940 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:08.615987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615925 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:08.615987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615928 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:08.616181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.615992 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:08.616181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616145 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.616282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616201 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy\") pod \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\" (UID: \"f789454e-6ef6-47d1-b92d-b18248f1ab9b\") " Oct 30 02:47:08.616687 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616664 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.616817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616693 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.616817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616708 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.616817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616723 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.616817 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.616738 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.617924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.617893 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "kube-api-access-7fvx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:08.617924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.617910 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:08.618066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.618027 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:08.618263 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.618248 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo" pod "f789454e-6ef6-47d1-b92d-b18248f1ab9b" (UID: "f789454e-6ef6-47d1-b92d-b18248f1ab9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:08.717363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.717335 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.717363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.717358 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.717363 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.717367 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fvx4\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-kube-api-access-7fvx4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:08.717575 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:08.717376 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f789454e-6ef6-47d1-b92d-b18248f1ab9b-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:09.468164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.468109 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" Oct 30 02:47:09.468164 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.468153 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg" event={"ID":"f789454e-6ef6-47d1-b92d-b18248f1ab9b","Type":"ContainerDied","Data":"467987cde086a732183814c264e29baef84bb3c4b033a9dc668d4e6738d22fb4"} Oct 30 02:47:09.468692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.468215 6275 scope.go:122] "RemoveContainer" containerID="6392b21109e286456f3f615a49c9c47bacba116912560c5c8bcea81ea9940e43" Oct 30 02:47:09.525354 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.525318 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:09.525676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.525656 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg"] Oct 30 02:47:09.529258 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.529236 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2819d2-openshift-default-5b6cdbf47-k9qvg"] Oct 30 02:47:09.740349 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.740270 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" path="/var/lib/kubelet/pods/f789454e-6ef6-47d1-b92d-b18248f1ab9b/volumes" Oct 30 02:47:09.834894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:09.834859 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr"] Oct 30 02:47:10.295371 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.295342 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:47:10.323014 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.322988 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:47:10.334625 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.334596 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume\") pod \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " Oct 30 02:47:10.335265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.334724 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j\") pod \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\" (UID: \"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c\") " Oct 30 02:47:10.335265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.335033 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume" pod "4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" (UID: "4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:10.335265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.335179 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:10.337298 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.337264 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j" pod "4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" (UID: "4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c"). InnerVolumeSpecName "kube-api-access-z574j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:10.435692 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.435657 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv\" (UniqueName: \"kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv\") pod \"c107c565-22f4-4184-bbc3-b60038674c5f\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " Oct 30 02:47:10.435911 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.435752 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume\") pod \"c107c565-22f4-4184-bbc3-b60038674c5f\" (UID: \"c107c565-22f4-4184-bbc3-b60038674c5f\") " Oct 30 02:47:10.436195 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.436160 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume" pod "c107c565-22f4-4184-bbc3-b60038674c5f" (UID: "c107c565-22f4-4184-bbc3-b60038674c5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:10.436328 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.436311 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107c565-22f4-4184-bbc3-b60038674c5f-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:10.436389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.436334 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c-kube-api-access-z574j\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:10.437837 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.437815 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv" pod "c107c565-22f4-4184-bbc3-b60038674c5f" (UID: "c107c565-22f4-4184-bbc3-b60038674c5f"). InnerVolumeSpecName "kube-api-access-lcxgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:10.477265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.477233 6275 generic.go:356] "Generic (PLEG): container finished" podID="c107c565-22f4-4184-bbc3-b60038674c5f" containerID="25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac" exitCode=0 Oct 30 02:47:10.477720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.477279 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" Oct 30 02:47:10.477720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.477308 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" event={"ID":"c107c565-22f4-4184-bbc3-b60038674c5f","Type":"ContainerDied","Data":"25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac"} Oct 30 02:47:10.477720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.477364 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj" event={"ID":"c107c565-22f4-4184-bbc3-b60038674c5f","Type":"ContainerDied","Data":"d06901bcd844caf6ccf84f8c9828e3baf97ddad95916d18bc42378566fd4cce8"} Oct 30 02:47:10.477720 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.477390 6275 scope.go:122] "RemoveContainer" containerID="25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac" Oct 30 02:47:10.479051 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.479025 6275 generic.go:356] "Generic (PLEG): container finished" podID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerID="92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3" exitCode=0 Oct 30 02:47:10.479181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.479055 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" event={"ID":"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c","Type":"ContainerDied","Data":"92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3"} Oct 30 02:47:10.479181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.479078 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" Oct 30 02:47:10.479181 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.479097 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng" event={"ID":"4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c","Type":"ContainerDied","Data":"c938ba155c87a71026b7d933bed350027dd948c18b20be7338b4f61b946738de"} Oct 30 02:47:10.487174 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.487153 6275 scope.go:122] "RemoveContainer" containerID="25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac" Oct 30 02:47:10.487474 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:10.487453 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac\": container with ID starting with 25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac not found: ID does not exist" containerID="25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac" Oct 30 02:47:10.487553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.487483 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac"} err="failed to get container status \"25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac\": rpc error: code = NotFound desc = could not find container \"25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac\": container with ID starting with 25ab88b1a545841ec9bfe46b14d7bd846e93f894fa31ccfb4c2b2e7fd34e3aac not found: ID does not exist" Oct 30 02:47:10.487553 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.487502 6275 scope.go:122] "RemoveContainer" containerID="92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3" Oct 30 02:47:10.496571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.496552 6275 scope.go:122] "RemoveContainer" containerID="92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3" Oct 30 02:47:10.496824 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:10.496807 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3\": container with ID starting with 92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3 not found: ID does not exist" containerID="92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3" Oct 30 02:47:10.496880 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.496832 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3"} err="failed to get container status \"92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3\": rpc error: code = NotFound desc = could not find container \"92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3\": container with ID starting with 92201f8315d84bf123a8d9a1a45ad97afecd25f68f6727d79360f12cebe08ac3 not found: ID does not exist" Oct 30 02:47:10.537223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.537132 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng"] Oct 30 02:47:10.537555 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.537536 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxgv\" (UniqueName: \"kubernetes.io/projected/c107c565-22f4-4184-bbc3-b60038674c5f-kube-api-access-lcxgv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:10.541235 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.541213 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01a9cd-86cc4f65f4-m8kng"] Oct 30 02:47:10.571858 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.571832 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj"] Oct 30 02:47:10.573981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:10.573955 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01a9cd-5c86c7f8dc-42ggj"] Oct 30 02:47:11.196795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.196315 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh"] Oct 30 02:47:11.196795 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.196570 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" podUID="34ef869d-3809-420e-9622-3fb670d7cf8b" containerName="maas-api" containerID="cri-o://f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a" gracePeriod=30 Oct 30 02:47:11.479972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.479944 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:47:11.488080 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.488048 6275 generic.go:356] "Generic (PLEG): container finished" podID="34ef869d-3809-420e-9622-3fb670d7cf8b" containerID="f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a" exitCode=0 Oct 30 02:47:11.488318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.488090 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" Oct 30 02:47:11.488318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.488142 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" event={"ID":"34ef869d-3809-420e-9622-3fb670d7cf8b","Type":"ContainerDied","Data":"f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a"} Oct 30 02:47:11.488318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.488201 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh" event={"ID":"34ef869d-3809-420e-9622-3fb670d7cf8b","Type":"ContainerDied","Data":"bfb9c98f9c9686e491a7a18ec60d8175d64fdb7d4bfe25bd468e12c702b1153e"} Oct 30 02:47:11.488318 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.488226 6275 scope.go:122] "RemoveContainer" containerID="f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a" Oct 30 02:47:11.500660 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.500640 6275 scope.go:122] "RemoveContainer" containerID="f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a" Oct 30 02:47:11.500946 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:11.500927 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a\": container with ID starting with f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a not found: ID does not exist" containerID="f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a" Oct 30 02:47:11.501015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.500958 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a"} err="failed to get container status \"f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a\": rpc error: code = NotFound desc = could not find container \"f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a\": container with ID starting with f9adc8d29bc568fb7b61e16092df3d0e04e7108af0f25d7590f4e9065b92e11a not found: ID does not exist" Oct 30 02:47:11.551013 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.550981 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca\") pod \"34ef869d-3809-420e-9622-3fb670d7cf8b\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " Oct 30 02:47:11.551175 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.551082 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2\" (UniqueName: \"kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2\") pod \"34ef869d-3809-420e-9622-3fb670d7cf8b\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " Oct 30 02:47:11.551236 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.551209 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs\") pod \"34ef869d-3809-420e-9622-3fb670d7cf8b\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " Oct 30 02:47:11.551284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.551253 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\") pod \"34ef869d-3809-420e-9622-3fb670d7cf8b\" (UID: \"34ef869d-3809-420e-9622-3fb670d7cf8b\") " Oct 30 02:47:11.551481 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.551452 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca" pod "34ef869d-3809-420e-9622-3fb670d7cf8b" (UID: "34ef869d-3809-420e-9622-3fb670d7cf8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:11.551721 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.551700 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ef869d-3809-420e-9622-3fb670d7cf8b-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:11.553505 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.553477 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls" pod "34ef869d-3809-420e-9622-3fb670d7cf8b" (UID: "34ef869d-3809-420e-9622-3fb670d7cf8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:11.553769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.553750 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs" pod "34ef869d-3809-420e-9622-3fb670d7cf8b" (UID: "34ef869d-3809-420e-9622-3fb670d7cf8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:11.554270 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.554240 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2" pod "34ef869d-3809-420e-9622-3fb670d7cf8b" (UID: "34ef869d-3809-420e-9622-3fb670d7cf8b"). InnerVolumeSpecName "kube-api-access-c7rj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:11.653587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.653490 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:11.653587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.653522 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ef869d-3809-420e-9622-3fb670d7cf8b-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:11.653587 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.653535 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7rj2\" (UniqueName: \"kubernetes.io/projected/34ef869d-3809-420e-9622-3fb670d7cf8b-kube-api-access-c7rj2\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:11.737716 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.737686 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" path="/var/lib/kubelet/pods/4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c/volumes" Oct 30 02:47:11.738086 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.738073 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" path="/var/lib/kubelet/pods/c107c565-22f4-4184-bbc3-b60038674c5f/volumes" Oct 30 02:47:11.823879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.823850 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh"] Oct 30 02:47:11.827300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:11.827273 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a1cd0-77bc54cbcb-p4mzh"] Oct 30 02:47:13.747171 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:13.747136 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ef869d-3809-420e-9622-3fb670d7cf8b" path="/var/lib/kubelet/pods/34ef869d-3809-420e-9622-3fb670d7cf8b/volumes" Oct 30 02:47:14.835480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:14.835434 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerName="payload-processing" containerID="cri-o://7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f" gracePeriod=25 Oct 30 02:47:15.477393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.477352 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" containerID="cri-o://6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696" gracePeriod=45 Oct 30 02:47:15.742907 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.742837 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 02:47:15.806894 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.806864 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn\" (UniqueName: \"kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn\") pod \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " Oct 30 02:47:15.807093 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.806938 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs\") pod \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " Oct 30 02:47:15.807093 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807010 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache\") pod \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " Oct 30 02:47:15.807228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807104 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm\") pod \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " Oct 30 02:47:15.807228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807195 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home\") pod \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " Oct 30 02:47:15.807228 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807225 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir\") pod \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\" (UID: \"c3b7b172-34fb-461b-ba7a-ae4a5c994600\") " Oct 30 02:47:15.807629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807555 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home" pod "c3b7b172-34fb-461b-ba7a-ae4a5c994600" (UID: "c3b7b172-34fb-461b-ba7a-ae4a5c994600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.807780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807627 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir" pod "c3b7b172-34fb-461b-ba7a-ae4a5c994600" (UID: "c3b7b172-34fb-461b-ba7a-ae4a5c994600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.807780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.807678 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache" pod "c3b7b172-34fb-461b-ba7a-ae4a5c994600" (UID: "c3b7b172-34fb-461b-ba7a-ae4a5c994600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.809462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.809436 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs" pod "c3b7b172-34fb-461b-ba7a-ae4a5c994600" (UID: "c3b7b172-34fb-461b-ba7a-ae4a5c994600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:15.809462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.809444 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn" pod "c3b7b172-34fb-461b-ba7a-ae4a5c994600" (UID: "c3b7b172-34fb-461b-ba7a-ae4a5c994600"). InnerVolumeSpecName "kube-api-access-m8mxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:15.809981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.809958 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm" pod "c3b7b172-34fb-461b-ba7a-ae4a5c994600" (UID: "c3b7b172-34fb-461b-ba7a-ae4a5c994600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.909232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.909194 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.909232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.909230 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.909232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.909239 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mxn\" (UniqueName: \"kubernetes.io/projected/c3b7b172-34fb-461b-ba7a-ae4a5c994600-kube-api-access-m8mxn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.909690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.909250 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b7b172-34fb-461b-ba7a-ae4a5c994600-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.909690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.909259 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.909690 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:15.909268 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b7b172-34fb-461b-ba7a-ae4a5c994600-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.518198 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.518159 6275 generic.go:356] "Generic (PLEG): container finished" podID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerID="6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696" exitCode=0 Oct 30 02:47:16.518477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.518205 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" Oct 30 02:47:16.518477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.518204 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" event={"ID":"c3b7b172-34fb-461b-ba7a-ae4a5c994600","Type":"ContainerDied","Data":"6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696"} Oct 30 02:47:16.518477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.518301 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" event={"ID":"c3b7b172-34fb-461b-ba7a-ae4a5c994600","Type":"ContainerDied","Data":"09af46c3f50fccd8b1d42e79d2f510d5efda093448f5952d0c057e72108fa147"} Oct 30 02:47:16.518477 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.518322 6275 scope.go:122] "RemoveContainer" containerID="6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696" Oct 30 02:47:16.530208 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.530183 6275 scope.go:122] "RemoveContainer" containerID="6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696" Oct 30 02:47:16.530520 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:16.530498 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696\": container with ID starting with 6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696 not found: ID does not exist" containerID="6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696" Oct 30 02:47:16.530589 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.530530 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696"} err="failed to get container status \"6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696\": rpc error: code = NotFound desc = could not find container \"6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696\": container with ID starting with 6099f064dc9b6922b1b3365d6e20d19f98235f5218ea5768d26631ee22296696 not found: ID does not exist" Oct 30 02:47:16.563641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.563606 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt"] Oct 30 02:47:16.574314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.574283 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt"] Oct 30 02:47:16.697735 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:16.697699 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:47:16.725525 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:16.725474 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-2-w2-719029-kserve-5b8b9bc5d8-xwtmt" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 02:47:17.738495 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:17.738457 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" path="/var/lib/kubelet/pods/c3b7b172-34fb-461b-ba7a-ae4a5c994600/volumes" Oct 30 02:47:18.515854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.515814 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:18.585566 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.585531 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg"] Oct 30 02:47:18.585785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.585756 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" podUID="4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" containerName="maas-api" containerID="cri-o://034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6" gracePeriod=30 Oct 30 02:47:18.868565 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.868538 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 02:47:18.944435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.944404 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs\") pod \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " Oct 30 02:47:18.944633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.944455 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca\") pod \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " Oct 30 02:47:18.944633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.944517 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls\") pod \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " Oct 30 02:47:18.944633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.944599 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv\" (UniqueName: \"kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv\") pod \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\" (UID: \"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda\") " Oct 30 02:47:18.944920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.944896 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca" pod "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" (UID: "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:18.945085 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.945066 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:18.946801 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.946771 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs" pod "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" (UID: "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:18.946899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.946831 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv" pod "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" (UID: "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda"). InnerVolumeSpecName "kube-api-access-rzpdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:18.947056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:18.947037 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls" pod "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" (UID: "4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:19.045920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.045878 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.045920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.045921 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpdv\" (UniqueName: \"kubernetes.io/projected/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-kube-api-access-rzpdv\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.046251 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.045934 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.450601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.450567 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:47:19.473233 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.473194 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:19.488601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.488577 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:47:19.539791 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.539756 6275 generic.go:356] "Generic (PLEG): container finished" podID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerID="82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201" exitCode=0 Oct 30 02:47:19.539791 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.539801 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" Oct 30 02:47:19.540057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.539834 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" event={"ID":"8cd3212b-a086-47bf-802f-ced6540a3d9b","Type":"ContainerDied","Data":"82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201"} Oct 30 02:47:19.540057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.539924 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm" event={"ID":"8cd3212b-a086-47bf-802f-ced6540a3d9b","Type":"ContainerDied","Data":"156c0a37c602a194d098820cd6073553631eef853152cc952c66c0d6b6437d9b"} Oct 30 02:47:19.540057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.539957 6275 scope.go:122] "RemoveContainer" containerID="82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201" Oct 30 02:47:19.544443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.544416 6275 generic.go:356] "Generic (PLEG): container finished" podID="4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" containerID="034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6" exitCode=0 Oct 30 02:47:19.544607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.544477 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" Oct 30 02:47:19.544607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.544506 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" event={"ID":"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda","Type":"ContainerDied","Data":"034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6"} Oct 30 02:47:19.544607 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.544563 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg" event={"ID":"4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda","Type":"ContainerDied","Data":"42e4f77bc26009621f3069cbab04440ae80bd3be5f9c69c10c64010da44001fa"} Oct 30 02:47:19.546646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.546620 6275 generic.go:356] "Generic (PLEG): container finished" podID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerID="139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88" exitCode=0 Oct 30 02:47:19.546800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.546669 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" Oct 30 02:47:19.546800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.546686 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" event={"ID":"5f9ff630-5b88-44db-af58-15ca56ed2389","Type":"ContainerDied","Data":"139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88"} Oct 30 02:47:19.546800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.546723 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc" event={"ID":"5f9ff630-5b88-44db-af58-15ca56ed2389","Type":"ContainerDied","Data":"b9df171a73175aba5dd0e30ad5c3ed1acd7716be00fa3e83ad86e32f7bdd9526"} Oct 30 02:47:19.550109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.550080 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt\" (UniqueName: \"kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt\") pod \"8cd3212b-a086-47bf-802f-ced6540a3d9b\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " Oct 30 02:47:19.550257 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.550238 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll\" (UniqueName: \"kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll\") pod \"5f9ff630-5b88-44db-af58-15ca56ed2389\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " Oct 30 02:47:19.550343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.550285 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume\") pod \"8cd3212b-a086-47bf-802f-ced6540a3d9b\" (UID: \"8cd3212b-a086-47bf-802f-ced6540a3d9b\") " Oct 30 02:47:19.550412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.550341 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume\") pod \"5f9ff630-5b88-44db-af58-15ca56ed2389\" (UID: \"5f9ff630-5b88-44db-af58-15ca56ed2389\") " Oct 30 02:47:19.551386 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.551096 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume" pod "5f9ff630-5b88-44db-af58-15ca56ed2389" (UID: "5f9ff630-5b88-44db-af58-15ca56ed2389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:19.551688 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.551654 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume" pod "8cd3212b-a086-47bf-802f-ced6540a3d9b" (UID: "8cd3212b-a086-47bf-802f-ced6540a3d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:19.556397 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.556369 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll" pod "5f9ff630-5b88-44db-af58-15ca56ed2389" (UID: "5f9ff630-5b88-44db-af58-15ca56ed2389"). InnerVolumeSpecName "kube-api-access-8cmll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:19.556580 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.556474 6275 scope.go:122] "RemoveContainer" containerID="82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201" Oct 30 02:47:19.557697 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.557674 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt" pod "8cd3212b-a086-47bf-802f-ced6540a3d9b" (UID: "8cd3212b-a086-47bf-802f-ced6540a3d9b"). InnerVolumeSpecName "kube-api-access-zrfpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:19.557819 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:19.557705 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201\": container with ID starting with 82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201 not found: ID does not exist" containerID="82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201" Oct 30 02:47:19.557819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.557739 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201"} err="failed to get container status \"82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201\": rpc error: code = NotFound desc = could not find container \"82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201\": container with ID starting with 82ede306ad7d190fe3ed4bccb14b02bac545d23de3cd34bc4fd82c6822ab8201 not found: ID does not exist" Oct 30 02:47:19.557819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.557763 6275 scope.go:122] "RemoveContainer" containerID="034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6" Oct 30 02:47:19.568827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.568809 6275 scope.go:122] "RemoveContainer" containerID="034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6" Oct 30 02:47:19.569116 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:19.569098 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6\": container with ID starting with 034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6 not found: ID does not exist" containerID="034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6" Oct 30 02:47:19.569182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.569135 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6"} err="failed to get container status \"034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6\": rpc error: code = NotFound desc = could not find container \"034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6\": container with ID starting with 034ae63fc5dcd552565cfd623c77865529d87f83c4965fd738ee9d70a19bf5d6 not found: ID does not exist" Oct 30 02:47:19.569182 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.569150 6275 scope.go:122] "RemoveContainer" containerID="139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88" Oct 30 02:47:19.575767 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:19.575744 6275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0940d0_a9fd_44cf_8ce5_2b1f8f1eeeda.slice\": RecentStats: unable to find data in memory cache]" Oct 30 02:47:19.581854 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.581821 6275 scope.go:122] "RemoveContainer" containerID="139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88" Oct 30 02:47:19.582365 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:19.582345 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88\": container with ID starting with 139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88 not found: ID does not exist" containerID="139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88" Oct 30 02:47:19.582454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.582370 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88"} err="failed to get container status \"139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88\": rpc error: code = NotFound desc = could not find container \"139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88\": container with ID starting with 139ecc9212b1134a510dbaa2022f51dee0aefc0a746bebb022c23bab6f935e88 not found: ID does not exist" Oct 30 02:47:19.616278 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.616257 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg"] Oct 30 02:47:19.620060 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.619981 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-719029-64d78488-qbspg"] Oct 30 02:47:19.651860 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.651835 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfpt\" (UniqueName: \"kubernetes.io/projected/8cd3212b-a086-47bf-802f-ced6540a3d9b-kube-api-access-zrfpt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.651939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.651861 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cmll\" (UniqueName: \"kubernetes.io/projected/5f9ff630-5b88-44db-af58-15ca56ed2389-kube-api-access-8cmll\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.651939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.651873 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd3212b-a086-47bf-802f-ced6540a3d9b-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.651939 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.651881 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9ff630-5b88-44db-af58-15ca56ed2389-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.738661 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.738575 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" path="/var/lib/kubelet/pods/4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda/volumes" Oct 30 02:47:19.875467 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.875429 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm"] Oct 30 02:47:19.879166 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.879139 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2819d2-6f8d85c6cb-bhhlm"] Oct 30 02:47:19.909431 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.909403 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc"] Oct 30 02:47:19.914480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:19.914451 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2819d2-6f9f68cc79-9fxrc"] Oct 30 02:47:21.737423 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:21.737389 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" path="/var/lib/kubelet/pods/5f9ff630-5b88-44db-af58-15ca56ed2389/volumes" Oct 30 02:47:21.737814 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:21.737781 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" path="/var/lib/kubelet/pods/8cd3212b-a086-47bf-802f-ced6540a3d9b/volumes" Oct 30 02:47:27.172502 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:27.172464 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8"] Oct 30 02:47:27.182711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:27.182681 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g"] Oct 30 02:47:29.473418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:29.473375 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:35.088302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.088276 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:47:35.221836 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.221746 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb\" (UniqueName: \"kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb\") pod \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " Oct 30 02:47:35.222006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.221919 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume\") pod \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\" (UID: \"1828b46c-1215-4b27-84a8-32d2e7f3f56b\") " Oct 30 02:47:35.222313 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.222289 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume" pod "1828b46c-1215-4b27-84a8-32d2e7f3f56b" (UID: "1828b46c-1215-4b27-84a8-32d2e7f3f56b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:35.223856 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.223830 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb" pod "1828b46c-1215-4b27-84a8-32d2e7f3f56b" (UID: "1828b46c-1215-4b27-84a8-32d2e7f3f56b"). InnerVolumeSpecName "kube-api-access-99dqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:35.322987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.322950 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dqb\" (UniqueName: \"kubernetes.io/projected/1828b46c-1215-4b27-84a8-32d2e7f3f56b-kube-api-access-99dqb\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:35.322987 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.322982 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1828b46c-1215-4b27-84a8-32d2e7f3f56b-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:35.645745 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.645712 6275 generic.go:356] "Generic (PLEG): container finished" podID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerID="7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f" exitCode=0 Oct 30 02:47:35.645951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.645781 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" event={"ID":"1828b46c-1215-4b27-84a8-32d2e7f3f56b","Type":"ContainerDied","Data":"7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f"} Oct 30 02:47:35.645951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.645810 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" Oct 30 02:47:35.645951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.645881 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr" event={"ID":"1828b46c-1215-4b27-84a8-32d2e7f3f56b","Type":"ContainerDied","Data":"74124afa233fbc03b4f9ce90bfb2c1ac09c467aca6684876aa29b70a67c69220"} Oct 30 02:47:35.645951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.645911 6275 scope.go:122] "RemoveContainer" containerID="7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f" Oct 30 02:47:35.656362 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.656345 6275 scope.go:122] "RemoveContainer" containerID="7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f" Oct 30 02:47:35.656619 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:35.656594 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f\": container with ID starting with 7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f not found: ID does not exist" containerID="7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f" Oct 30 02:47:35.656668 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.656634 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f"} err="failed to get container status \"7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f\": rpc error: code = NotFound desc = could not find container \"7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f\": container with ID starting with 7e57d827b15c64490ea36eef7d1586eec9a18c380aca2d3327b6a4a25cd3255f not found: ID does not exist" Oct 30 02:47:35.695577 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.695547 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr"] Oct 30 02:47:35.705908 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.705884 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9a1cd0-76568b9b9d-mg2lr"] Oct 30 02:47:35.737561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:35.737529 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" path="/var/lib/kubelet/pods/1828b46c-1215-4b27-84a8-32d2e7f3f56b/volumes" Oct 30 02:47:40.840266 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:40.840231 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8"] Oct 30 02:47:40.840711 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:40.840420 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" podUID="9e176405-a61d-4763-8f40-1c40de877bb2" containerName="maas-api" containerID="cri-o://c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c" gracePeriod=30 Oct 30 02:47:41.108156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.108091 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 02:47:41.175036 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.175002 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9\" (UniqueName: \"kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9\") pod \"9e176405-a61d-4763-8f40-1c40de877bb2\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " Oct 30 02:47:41.175271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.175050 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs\") pod \"9e176405-a61d-4763-8f40-1c40de877bb2\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " Oct 30 02:47:41.175271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.175071 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca\") pod \"9e176405-a61d-4763-8f40-1c40de877bb2\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " Oct 30 02:47:41.175271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.175186 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls\") pod \"9e176405-a61d-4763-8f40-1c40de877bb2\" (UID: \"9e176405-a61d-4763-8f40-1c40de877bb2\") " Oct 30 02:47:41.175623 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.175593 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca" pod "9e176405-a61d-4763-8f40-1c40de877bb2" (UID: "9e176405-a61d-4763-8f40-1c40de877bb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:41.178781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.177813 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9" pod "9e176405-a61d-4763-8f40-1c40de877bb2" (UID: "9e176405-a61d-4763-8f40-1c40de877bb2"). InnerVolumeSpecName "kube-api-access-5wzn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:41.178781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.178019 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs" pod "9e176405-a61d-4763-8f40-1c40de877bb2" (UID: "9e176405-a61d-4763-8f40-1c40de877bb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:41.179479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.179433 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls" pod "9e176405-a61d-4763-8f40-1c40de877bb2" (UID: "9e176405-a61d-4763-8f40-1c40de877bb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:41.276614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.276575 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.276614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.276609 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzn9\" (UniqueName: \"kubernetes.io/projected/9e176405-a61d-4763-8f40-1c40de877bb2-kube-api-access-5wzn9\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.276614 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.276619 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e176405-a61d-4763-8f40-1c40de877bb2-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.276864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.276628 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e176405-a61d-4763-8f40-1c40de877bb2-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.690245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.683668 6275 generic.go:356] "Generic (PLEG): container finished" podID="9e176405-a61d-4763-8f40-1c40de877bb2" containerID="c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c" exitCode=0 Oct 30 02:47:41.690245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.683734 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" Oct 30 02:47:41.690245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.683713 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" event={"ID":"9e176405-a61d-4763-8f40-1c40de877bb2","Type":"ContainerDied","Data":"c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c"} Oct 30 02:47:41.690245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.683833 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8" event={"ID":"9e176405-a61d-4763-8f40-1c40de877bb2","Type":"ContainerDied","Data":"87ea6717e2cbd393427cf015c97ba870aa74bb16e35fbf830acd6f5e1b23c812"} Oct 30 02:47:41.690245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.683872 6275 scope.go:122] "RemoveContainer" containerID="c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c" Oct 30 02:47:41.694213 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.694189 6275 scope.go:122] "RemoveContainer" containerID="c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c" Oct 30 02:47:41.694512 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:41.694492 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c\": container with ID starting with c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c not found: ID does not exist" containerID="c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c" Oct 30 02:47:41.694596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.694525 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c"} err="failed to get container status \"c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c\": rpc error: code = NotFound desc = could not find container \"c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c\": container with ID starting with c66f08182b92e1d978ba9b591bab0f0e2d8ebd19865fff3af03af7ca3877490c not found: ID does not exist" Oct 30 02:47:41.731089 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.731055 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8"] Oct 30 02:47:41.741616 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:41.741582 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-78d5ff-85747cfc6f-6bqm8"] Oct 30 02:47:42.194792 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.194741 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" containerID="cri-o://55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0" gracePeriod=45 Oct 30 02:47:42.203263 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.203069 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" containerID="cri-o://624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473" gracePeriod=45 Oct 30 02:47:42.353160 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.351305 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": dial tcp 10.134.0.67:8000: connect: connection refused" Oct 30 02:47:42.515586 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.515560 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 02:47:42.517644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.517621 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607231 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir\") pod \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607294 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs\" (UniqueName: \"kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs\") pod \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607348 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm\") pod \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607432 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache\") pod \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607502 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir\") pod \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607588 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k\" (UniqueName: \"kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k\") pod \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607620 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs\") pod \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607676 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache\") pod \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607709 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home\") pod \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607748 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m\" (UniqueName: \"kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m\") pod \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\" (UID: \"de2f5ccc-398f-443a-a4b4-891bbaae9b77\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607777 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home\") pod \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " Oct 30 02:47:42.608214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.607812 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm\") pod \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\" (UID: \"daf72a41-7c0d-4851-9fc9-57de3e8a42d3\") " Oct 30 02:47:42.610150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.609627 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home" pod "daf72a41-7c0d-4851-9fc9-57de3e8a42d3" (UID: "daf72a41-7c0d-4851-9fc9-57de3e8a42d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.610150 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.609977 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir" pod "de2f5ccc-398f-443a-a4b4-891bbaae9b77" (UID: "de2f5ccc-398f-443a-a4b4-891bbaae9b77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.610373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.610239 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home" pod "de2f5ccc-398f-443a-a4b4-891bbaae9b77" (UID: "de2f5ccc-398f-443a-a4b4-891bbaae9b77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.610373 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.610249 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache" pod "daf72a41-7c0d-4851-9fc9-57de3e8a42d3" (UID: "daf72a41-7c0d-4851-9fc9-57de3e8a42d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.610492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.610450 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache" pod "de2f5ccc-398f-443a-a4b4-891bbaae9b77" (UID: "de2f5ccc-398f-443a-a4b4-891bbaae9b77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.610492 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.610455 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir" pod "daf72a41-7c0d-4851-9fc9-57de3e8a42d3" (UID: "daf72a41-7c0d-4851-9fc9-57de3e8a42d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.614159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.612442 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm" pod "daf72a41-7c0d-4851-9fc9-57de3e8a42d3" (UID: "daf72a41-7c0d-4851-9fc9-57de3e8a42d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.614159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.612708 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm" pod "de2f5ccc-398f-443a-a4b4-891bbaae9b77" (UID: "de2f5ccc-398f-443a-a4b4-891bbaae9b77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.614159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.613246 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs" pod "daf72a41-7c0d-4851-9fc9-57de3e8a42d3" (UID: "daf72a41-7c0d-4851-9fc9-57de3e8a42d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:42.614159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.613310 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k" pod "daf72a41-7c0d-4851-9fc9-57de3e8a42d3" (UID: "daf72a41-7c0d-4851-9fc9-57de3e8a42d3"). InnerVolumeSpecName "kube-api-access-mmz6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:42.614159 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.613367 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m" pod "de2f5ccc-398f-443a-a4b4-891bbaae9b77" (UID: "de2f5ccc-398f-443a-a4b4-891bbaae9b77"). InnerVolumeSpecName "kube-api-access-jcc5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:42.614535 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.614507 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs" pod "de2f5ccc-398f-443a-a4b4-891bbaae9b77" (UID: "de2f5ccc-398f-443a-a4b4-891bbaae9b77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:42.693197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.693111 6275 generic.go:356] "Generic (PLEG): container finished" podID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerID="624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473" exitCode=0 Oct 30 02:47:42.693404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.693262 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" event={"ID":"daf72a41-7c0d-4851-9fc9-57de3e8a42d3","Type":"ContainerDied","Data":"624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473"} Oct 30 02:47:42.693404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.693300 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" event={"ID":"daf72a41-7c0d-4851-9fc9-57de3e8a42d3","Type":"ContainerDied","Data":"86c7f11c7e84f1ee237d715381f77c8427432069e79a7a852599b1e0624fccc1"} Oct 30 02:47:42.693404 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.693324 6275 scope.go:122] "RemoveContainer" containerID="624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473" Oct 30 02:47:42.693676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.693439 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g" Oct 30 02:47:42.703151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.702338 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" Oct 30 02:47:42.707864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.705188 6275 generic.go:356] "Generic (PLEG): container finished" podID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerID="55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0" exitCode=0 Oct 30 02:47:42.707864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.705294 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" event={"ID":"de2f5ccc-398f-443a-a4b4-891bbaae9b77","Type":"ContainerDied","Data":"55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0"} Oct 30 02:47:42.707864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.705330 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8" event={"ID":"de2f5ccc-398f-443a-a4b4-891bbaae9b77","Type":"ContainerDied","Data":"1b26613c34a49c5674af69d30433074b7abf9c1ace65fd3e26d429f02ae9a662"} Oct 30 02:47:42.709074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709054 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709074 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709076 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2f5ccc-398f-443a-a4b4-891bbaae9b77-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709083 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709091 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709099 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709107 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmz6k\" (UniqueName: \"kubernetes.io/projected/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-kube-api-access-mmz6k\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709131 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709146 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709157 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2f5ccc-398f-443a-a4b4-891bbaae9b77-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709170 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcc5m\" (UniqueName: \"kubernetes.io/projected/de2f5ccc-398f-443a-a4b4-891bbaae9b77-kube-api-access-jcc5m\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709182 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.709254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.709195 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf72a41-7c0d-4851-9fc9-57de3e8a42d3-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.713641 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.713617 6275 scope.go:122] "RemoveContainer" containerID="624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473" Oct 30 02:47:42.714016 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:42.713951 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473\": container with ID starting with 624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473 not found: ID does not exist" containerID="624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473" Oct 30 02:47:42.714016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.713989 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473"} err="failed to get container status \"624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473\": rpc error: code = NotFound desc = could not find container \"624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473\": container with ID starting with 624b1aaad93f4952b56578098239ff57fe40f48f5785c52c71f4d5137c586473 not found: ID does not exist" Oct 30 02:47:42.714016 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.714012 6275 scope.go:122] "RemoveContainer" containerID="55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0" Oct 30 02:47:42.725891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.725867 6275 scope.go:122] "RemoveContainer" containerID="55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0" Oct 30 02:47:42.726314 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:42.726264 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0\": container with ID starting with 55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0 not found: ID does not exist" containerID="55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0" Oct 30 02:47:42.726389 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.726302 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0"} err="failed to get container status \"55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0\": rpc error: code = NotFound desc = could not find container \"55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0\": container with ID starting with 55cd2e05071d4804d75ae40ce6753186fa03e74d701c4d4de154bc6b57ad75d0 not found: ID does not exist" Oct 30 02:47:42.789533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.789483 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8"] Oct 30 02:47:42.799376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.799339 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-w0-c40a89-kserve-6d56fb7698-dn7c8"] Oct 30 02:47:42.874443 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.874402 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g"] Oct 30 02:47:42.893790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:42.893755 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/premium-w0-c40a89-kserve-6d9c5b5899-ncd8g"] Oct 30 02:47:43.736557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:43.736527 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e176405-a61d-4763-8f40-1c40de877bb2" path="/var/lib/kubelet/pods/9e176405-a61d-4763-8f40-1c40de877bb2/volumes" Oct 30 02:47:43.736956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:43.736943 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" path="/var/lib/kubelet/pods/daf72a41-7c0d-4851-9fc9-57de3e8a42d3/volumes" Oct 30 02:47:43.737339 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:43.737326 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" path="/var/lib/kubelet/pods/de2f5ccc-398f-443a-a4b4-891bbaae9b77/volumes" Oct 30 02:47:46.696068 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:46.696028 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:47:46.745613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:46.745582 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2"] Oct 30 02:47:46.745835 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:46.745810 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" podUID="2cbc2cea-aff0-4325-9302-3605211bca38" containerName="maas-api" containerID="cri-o://dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537" gracePeriod=30 Oct 30 02:47:46.997799 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:46.997774 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 02:47:47.055110 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.055076 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca\") pod \"2cbc2cea-aff0-4325-9302-3605211bca38\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " Oct 30 02:47:47.055304 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.055165 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls\") pod \"2cbc2cea-aff0-4325-9302-3605211bca38\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " Oct 30 02:47:47.055304 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.055277 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt\" (UniqueName: \"kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt\") pod \"2cbc2cea-aff0-4325-9302-3605211bca38\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " Oct 30 02:47:47.055406 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.055395 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs\") pod \"2cbc2cea-aff0-4325-9302-3605211bca38\" (UID: \"2cbc2cea-aff0-4325-9302-3605211bca38\") " Oct 30 02:47:47.055559 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.055536 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca" pod "2cbc2cea-aff0-4325-9302-3605211bca38" (UID: "2cbc2cea-aff0-4325-9302-3605211bca38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:47.055813 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.055793 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbc2cea-aff0-4325-9302-3605211bca38-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:47.057427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.057382 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs" pod "2cbc2cea-aff0-4325-9302-3605211bca38" (UID: "2cbc2cea-aff0-4325-9302-3605211bca38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:47.057539 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.057512 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt" pod "2cbc2cea-aff0-4325-9302-3605211bca38" (UID: "2cbc2cea-aff0-4325-9302-3605211bca38"). InnerVolumeSpecName "kube-api-access-hhlvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:47.057597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.057574 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls" pod "2cbc2cea-aff0-4325-9302-3605211bca38" (UID: "2cbc2cea-aff0-4325-9302-3605211bca38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:47.156460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.156428 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:47.156460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.156455 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbc2cea-aff0-4325-9302-3605211bca38-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:47.156460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.156465 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlvt\" (UniqueName: \"kubernetes.io/projected/2cbc2cea-aff0-4325-9302-3605211bca38-kube-api-access-hhlvt\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:47.738403 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.738363 6275 generic.go:356] "Generic (PLEG): container finished" podID="2cbc2cea-aff0-4325-9302-3605211bca38" containerID="dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537" exitCode=0 Oct 30 02:47:47.738954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.738414 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" Oct 30 02:47:47.738954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.738452 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" event={"ID":"2cbc2cea-aff0-4325-9302-3605211bca38","Type":"ContainerDied","Data":"dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537"} Oct 30 02:47:47.738954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.738507 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2" event={"ID":"2cbc2cea-aff0-4325-9302-3605211bca38","Type":"ContainerDied","Data":"c4db29e51b7075b7863a27e67ff29932621605854b661293393f7b8cab414f1b"} Oct 30 02:47:47.738954 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.738530 6275 scope.go:122] "RemoveContainer" containerID="dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537" Oct 30 02:47:47.749156 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.748948 6275 scope.go:122] "RemoveContainer" containerID="dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537" Oct 30 02:47:47.749459 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:47.749436 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537\": container with ID starting with dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537 not found: ID does not exist" containerID="dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537" Oct 30 02:47:47.749544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.749465 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537"} err="failed to get container status \"dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537\": rpc error: code = NotFound desc = could not find container \"dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537\": container with ID starting with dcc724e64636194be4a4c329462b3b6e99cf839a3fa5567688e1bb68692fb537 not found: ID does not exist" Oct 30 02:47:47.814714 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.814679 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2"] Oct 30 02:47:47.852351 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:47.852324 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c40a89-678dfcf565-24ln2"] Oct 30 02:47:49.737677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:49.737640 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbc2cea-aff0-4325-9302-3605211bca38" path="/var/lib/kubelet/pods/2cbc2cea-aff0-4325-9302-3605211bca38/volumes" Oct 30 02:47:52.963541 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:52.963487 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4"] Oct 30 02:47:52.964077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:52.963765 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" containerID="cri-o://5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2" gracePeriod=30 Oct 30 02:47:54.007286 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:54.007252 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7795bf7868-gmhpg_09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3/authorino/0.log" Oct 30 02:47:55.471675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:55.471637 6275 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:47:55.472176 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:55.471698 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:47:56.571427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:56.571395 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 02:47:57.827863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:57.827823 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-7wgng_400efcf6-470e-4589-9ad0-fb0c4458116b/manager/0.log" Oct 30 02:47:58.220945 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.220917 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 02:47:58.369463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369380 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369420 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369480 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369534 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369558 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369577 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369607 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.369706 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369678 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.370056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369743 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs\") pod \"bd6b46a2-a560-487b-b2f9-c456278bb868\" (UID: \"bd6b46a2-a560-487b-b2f9-c456278bb868\") " Oct 30 02:47:58.370056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369879 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:58.370056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369897 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:58.370056 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.369953 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:58.370324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370074 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:58.370324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370166 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:58.370324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370287 6275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-credential-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.370324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370307 6275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-socket\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.370324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370323 6275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-data\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.370518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370336 6275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-workload-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.370518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.370349 6275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6b46a2-a560-487b-b2f9-c456278bb868-istiod-ca-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.371991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.371960 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "kube-api-access-dtcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:58.372172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.372009 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:58.372245 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.372221 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:58.372291 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.372245 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token" pod "bd6b46a2-a560-487b-b2f9-c456278bb868" (UID: "bd6b46a2-a560-487b-b2f9-c456278bb868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:58.446281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.446244 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 02:47:58.470884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.470854 6275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-envoy\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.470884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.470880 6275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-token\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.470884 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.470889 6275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6b46a2-a560-487b-b2f9-c456278bb868-istio-podinfo\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.471169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.470898 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtcs2\" (UniqueName: \"kubernetes.io/projected/bd6b46a2-a560-487b-b2f9-c456278bb868-kube-api-access-dtcs2\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:47:58.792927 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.792894 6275 generic.go:356] "Generic (PLEG): container finished" podID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerID="5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2" exitCode=0 Oct 30 02:47:58.793169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.792941 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" Oct 30 02:47:58.793169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.792956 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" event={"ID":"bd6b46a2-a560-487b-b2f9-c456278bb868","Type":"ContainerDied","Data":"5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2"} Oct 30 02:47:58.793169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.793004 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4" event={"ID":"bd6b46a2-a560-487b-b2f9-c456278bb868","Type":"ContainerDied","Data":"053232f981cf0da15167bafbac8fe1c5fed623347b5bcb0a6e15138a5d93d6fe"} Oct 30 02:47:58.793169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.793031 6275 scope.go:122] "RemoveContainer" containerID="5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2" Oct 30 02:47:58.804092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.804071 6275 scope.go:122] "RemoveContainer" containerID="5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2" Oct 30 02:47:58.804396 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:47:58.804377 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2\": container with ID starting with 5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2 not found: ID does not exist" containerID="5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2" Oct 30 02:47:58.804457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.804405 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2"} err="failed to get container status \"5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2\": rpc error: code = NotFound desc = could not find container \"5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2\": container with ID starting with 5fdd85fe6b6dcf691f1028e8c71f70ddfe2af9447340593fd28a42b47510b5e2 not found: ID does not exist" Oct 30 02:47:58.839504 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.839474 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4"] Oct 30 02:47:58.850434 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:58.850410 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-s99t4"] Oct 30 02:47:59.737012 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:47:59.736973 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" path="/var/lib/kubelet/pods/bd6b46a2-a560-487b-b2f9-c456278bb868/volumes" Oct 30 02:48:07.568453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568415 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm"] Oct 30 02:48:07.568859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568834 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.568859 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568849 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568867 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" containerName="maas-api" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568873 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" containerName="maas-api" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568882 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568887 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568898 6275 cpu_manager.go:414] "containerMap: removing container" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerName="payload-processing" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568903 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerName="payload-processing" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568911 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerName="payload-processing" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568915 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerName="payload-processing" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568925 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568932 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.568941 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568941 6275 cpu_manager.go:414] "containerMap: removing container" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568949 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568963 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568970 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568980 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a1f64997-cd28-4d25-ad03-ddd332ac48fb" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568987 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f64997-cd28-4d25-ad03-ddd332ac48fb" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.568995 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569002 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569015 6275 cpu_manager.go:414] "containerMap: removing container" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569023 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569034 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569039 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569045 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e82f90c0-f1a2-46b7-8dd5-117b35481b71" containerName="revoke-keys" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569049 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82f90c0-f1a2-46b7-8dd5-117b35481b71" containerName="revoke-keys" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569058 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569064 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569072 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c564e47b-d931-4e9f-a58e-00d3f9d314a9" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569077 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c564e47b-d931-4e9f-a58e-00d3f9d314a9" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569087 6275 cpu_manager.go:414] "containerMap: removing container" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569094 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569108 6275 cpu_manager.go:414] "containerMap: removing container" podUID="236d9bfe-9a3f-4240-80a6-c01b604d31b0" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569115 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236d9bfe-9a3f-4240-80a6-c01b604d31b0" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569140 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9e176405-a61d-4763-8f40-1c40de877bb2" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569147 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e176405-a61d-4763-8f40-1c40de877bb2" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569156 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569163 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569173 6275 cpu_manager.go:414] "containerMap: removing container" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569180 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569189 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569196 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569210 6275 cpu_manager.go:414] "containerMap: removing container" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569217 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569225 6275 cpu_manager.go:414] "containerMap: removing container" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569231 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569247 6275 cpu_manager.go:414] "containerMap: removing container" podUID="37349c06-fce3-45a9-9128-344fa1508a06" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569253 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37349c06-fce3-45a9-9128-344fa1508a06" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569264 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569270 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569284 6275 cpu_manager.go:414] "containerMap: removing container" podUID="2cbc2cea-aff0-4325-9302-3605211bca38" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569290 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbc2cea-aff0-4325-9302-3605211bca38" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569299 6275 cpu_manager.go:414] "containerMap: removing container" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569306 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569315 6275 cpu_manager.go:414] "containerMap: removing container" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569321 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569330 6275 cpu_manager.go:414] "containerMap: removing container" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569337 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569345 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569351 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569365 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569373 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569385 6275 cpu_manager.go:414] "containerMap: removing container" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569393 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerName="payload-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569402 6275 cpu_manager.go:414] "containerMap: removing container" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569408 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569415 6275 cpu_manager.go:414] "containerMap: removing container" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569419 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569427 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569432 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerName="payload-pre-processing" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569441 6275 cpu_manager.go:414] "containerMap: removing container" podUID="34ef869d-3809-420e-9622-3fb670d7cf8b" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569445 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ef869d-3809-420e-9622-3fb670d7cf8b" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569455 6275 cpu_manager.go:414] "containerMap: removing container" podUID="4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569460 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" containerName="maas-api" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569469 6275 cpu_manager.go:414] "containerMap: removing container" podUID="bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569477 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569492 6275 cpu_manager.go:414] "containerMap: removing container" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569499 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" Oct 30 02:48:07.569463 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569510 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d8ccdb02-96b3-4b75-822c-e69ddfa45793" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569515 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ccdb02-96b3-4b75-822c-e69ddfa45793" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569606 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb67639-18bc-4b31-8ff8-dde4cdeb81ab" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569616 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569622 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="01bdad21-9f15-43a1-8af7-1e22cf49aaa9" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569628 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a372f44-cdb5-4db0-b3c4-fe0e3e32bd4c" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569636 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569644 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="369285b9-8086-43a0-a3df-36de41f1ea34" containerName="payload-pre-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569651 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569656 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f64997-cd28-4d25-ad03-ddd332ac48fb" containerName="main" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569663 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569668 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ce7a1c-cc4e-4576-b408-a3ea63a88c1e" containerName="payload-pre-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569674 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5a4a6d-5e8e-480e-a35d-1c339fb4a69a" containerName="main" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569679 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82f90c0-f1a2-46b7-8dd5-117b35481b71" containerName="revoke-keys" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569684 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569688 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c564e47b-d931-4e9f-a58e-00d3f9d314a9" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569693 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd3212b-a086-47bf-802f-ced6540a3d9b" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569699 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="236d9bfe-9a3f-4240-80a6-c01b604d31b0" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569707 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e176405-a61d-4763-8f40-1c40de877bb2" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569716 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6b46a2-a560-487b-b2f9-c456278bb868" containerName="istio-proxy" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569724 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1828b46c-1215-4b27-84a8-32d2e7f3f56b" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569731 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2f5ccc-398f-443a-a4b4-891bbaae9b77" containerName="main" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569735 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7413499b-ba8b-4489-bc32-2655b26ab793" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569741 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="37349c06-fce3-45a9-9128-344fa1508a06" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569745 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569751 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbc2cea-aff0-4325-9302-3605211bca38" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569756 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d45a619-bd39-48ff-84d2-c23e59556508" containerName="cleanup" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569762 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf72a41-7c0d-4851-9fc9-57de3e8a42d3" containerName="main" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569767 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c955d1c-d428-4ac7-b931-4cfe83e3ca92" containerName="istio-proxy" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569773 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c107c565-22f4-4184-bbc3-b60038674c5f" containerName="payload-pre-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569778 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e530cbf4-a38c-4a03-a242-948d62ece7d7" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569782 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6fe799-f177-4de5-b3d1-be217af32a1e" containerName="payload-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569789 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="310a8418-7f15-4283-b7b7-ebf83096d94c" containerName="payload-pre-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569797 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d23e69-56ba-4550-89c9-d66cc6a895ed" containerName="payload-pre-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569806 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9ff630-5b88-44db-af58-15ca56ed2389" containerName="payload-pre-processing" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569812 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ef869d-3809-420e-9622-3fb670d7cf8b" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569817 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0940d0-a9fd-44cf-8ce5-2b1f8f1eeeda" containerName="maas-api" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569822 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf70cf0e-c1ec-42dc-a4a7-244d89d835ac" containerName="main" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569827 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b7b172-34fb-461b-ba7a-ae4a5c994600" containerName="main" Oct 30 02:48:07.571557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.569832 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ccdb02-96b3-4b75-822c-e69ddfa45793" containerName="payload-processing" Oct 30 02:48:07.574462 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.574440 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.589096 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.589071 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a0944980-serving-cert\"" Oct 30 02:48:07.601688 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.601650 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm"] Oct 30 02:48:07.666090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.666061 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.666310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.666097 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.666310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.666248 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.666310 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.666286 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxxc\" (UniqueName: \"kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.767368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.767333 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.767578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.767387 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxxc\" (UniqueName: \"kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.767578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.767448 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.767578 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.767463 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.768254 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.768230 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.770032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.770000 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.770172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.770073 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.800925 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.800888 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxxc\" (UniqueName: \"kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc\") pod \"maas-api-e2e-ait-a0944980-9876c99b5-mshzm\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:07.888283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:07.888203 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:08.048831 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.048787 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk"] Oct 30 02:48:08.049273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.049246 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" Oct 30 02:48:08.049273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.049274 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" Oct 30 02:48:08.049401 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.049390 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d2ac4c-10bf-47e7-bff0-254447ab0b62" containerName="cleanup" Oct 30 02:48:08.049435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.049401 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f789454e-6ef6-47d1-b92d-b18248f1ab9b" containerName="istio-proxy" Oct 30 02:48:08.049435 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.049409 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c06325f-3df6-48c3-8f02-53aeef8a2de2" containerName="istio-proxy" Oct 30 02:48:08.053588 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.053567 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.070066 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.070037 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.070232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.070105 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cb7\" (UniqueName: \"kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7\") pod \"payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.078336 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.078310 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk"] Oct 30 02:48:08.139556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.139463 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0944980\"" Oct 30 02:48:08.139556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.139515 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a0944980-dockercfg-bbgjq\"" Oct 30 02:48:08.171420 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.171386 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.171611 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.171552 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cb7\" (UniqueName: \"kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7\") pod \"payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.172828 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.172801 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.193654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.193622 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cb7\" (UniqueName: \"kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7\") pod \"payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.209112 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.209083 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm"] Oct 30 02:48:08.213324 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:48:08.213298 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e4f4ed_03e2_4849_9d0a_4cc9a2caf7a4.slice/crio-6541fb1f7f6fae02fe9d06c6330258b5defafeba78cc6ca10d8ed8a501c45e2a WatchSource:0}: Error finding container 6541fb1f7f6fae02fe9d06c6330258b5defafeba78cc6ca10d8ed8a501c45e2a: Status 404 returned error can't find the container with id 6541fb1f7f6fae02fe9d06c6330258b5defafeba78cc6ca10d8ed8a501c45e2a Oct 30 02:48:08.215151 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.215107 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:48:08.367250 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.367215 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.681142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.681039 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk"] Oct 30 02:48:08.684929 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:48:08.684899 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05576d1_f3d8_4966_aab1_50743855e74c.slice/crio-6f68cd4c83a273250a03c4d23eee40cc72bc9110d837462c0697e561702b962b WatchSource:0}: Error finding container 6f68cd4c83a273250a03c4d23eee40cc72bc9110d837462c0697e561702b962b: Status 404 returned error can't find the container with id 6f68cd4c83a273250a03c4d23eee40cc72bc9110d837462c0697e561702b962b Oct 30 02:48:08.847468 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.847437 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" event={"ID":"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4","Type":"ContainerStarted","Data":"6541fb1f7f6fae02fe9d06c6330258b5defafeba78cc6ca10d8ed8a501c45e2a"} Oct 30 02:48:08.849317 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.849285 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" event={"ID":"e05576d1-f3d8-4966-aab1-50743855e74c","Type":"ContainerStarted","Data":"8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548"} Oct 30 02:48:08.849424 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.849344 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" event={"ID":"e05576d1-f3d8-4966-aab1-50743855e74c","Type":"ContainerStarted","Data":"6f68cd4c83a273250a03c4d23eee40cc72bc9110d837462c0697e561702b962b"} Oct 30 02:48:08.849518 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.849501 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:08.892608 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:08.892554 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" podStartSLOduration=0.892540111 podStartE2EDuration="892.540111ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:08.891957553 +0000 UTC m=+4717.283765186" watchObservedRunningTime="2026-10-30 02:48:08.892540111 +0000 UTC m=+4717.284347745" Oct 30 02:48:09.856807 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:09.856763 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" event={"ID":"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4","Type":"ContainerStarted","Data":"537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb"} Oct 30 02:48:09.857376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:09.857011 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:09.897377 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:09.897327 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" podStartSLOduration=2.307184658 podStartE2EDuration="2.897314472s" totalImagesPullingTime="590.129814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:09.895999292 +0000 UTC m=+4718.287806928" watchObservedRunningTime="2026-10-30 02:48:09.897314472 +0000 UTC m=+4718.289122105" Oct 30 02:48:15.868957 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:15.868925 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:16.696076 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:48:16.696037 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:48:19.860302 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:19.860275 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:32.460677 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.460642 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk"] Oct 30 02:48:32.620544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.620504 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm"] Oct 30 02:48:32.620776 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.620719 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" podUID="62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" containerName="maas-api" containerID="cri-o://537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb" gracePeriod=30 Oct 30 02:48:32.873684 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.873659 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:32.968899 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.968862 6275 generic.go:356] "Generic (PLEG): container finished" podID="62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" containerID="537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb" exitCode=0 Oct 30 02:48:32.969087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.968913 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" Oct 30 02:48:32.969087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.968942 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" event={"ID":"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4","Type":"ContainerDied","Data":"537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb"} Oct 30 02:48:32.969087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.968998 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm" event={"ID":"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4","Type":"ContainerDied","Data":"6541fb1f7f6fae02fe9d06c6330258b5defafeba78cc6ca10d8ed8a501c45e2a"} Oct 30 02:48:32.969087 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.969029 6275 scope.go:122] "RemoveContainer" containerID="537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb" Oct 30 02:48:32.977991 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.977962 6275 scope.go:122] "RemoveContainer" containerID="537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb" Oct 30 02:48:32.978332 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:48:32.978312 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb\": container with ID starting with 537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb not found: ID does not exist" containerID="537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb" Oct 30 02:48:32.978418 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:32.978347 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb"} err="failed to get container status \"537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb\": rpc error: code = NotFound desc = could not find container \"537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb\": container with ID starting with 537b784aaa71eb9cf6c347c33e5e4b1da7a6b003682aedc4ec9681b2ea2ffbeb not found: ID does not exist" Oct 30 02:48:33.022729 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.022684 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca\") pod \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " Oct 30 02:48:33.022916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.022735 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs\") pod \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " Oct 30 02:48:33.022916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.022774 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls\") pod \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " Oct 30 02:48:33.022916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.022817 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc\" (UniqueName: \"kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc\") pod \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\" (UID: \"62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4\") " Oct 30 02:48:33.028372 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.028338 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca" pod "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" (UID: "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:48:33.029705 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.029658 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc" pod "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" (UID: "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4"). InnerVolumeSpecName "kube-api-access-2nxxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:33.034253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.034211 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs" pod "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" (UID: "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:48:33.043919 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.043886 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls" pod "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" (UID: "62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:48:33.124847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.124811 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:33.124847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.124840 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:33.124847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.124853 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:33.125079 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.124865 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxxc\" (UniqueName: \"kubernetes.io/projected/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4-kube-api-access-2nxxc\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:33.325342 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.325309 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm"] Oct 30 02:48:33.335387 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.335358 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0944980-9876c99b5-mshzm"] Oct 30 02:48:33.736713 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:33.736635 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" path="/var/lib/kubelet/pods/62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4/volumes" Oct 30 02:48:37.461246 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.461186 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" podUID="e05576d1-f3d8-4966-aab1-50743855e74c" containerName="payload-processing" containerID="cri-o://8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548" gracePeriod=25 Oct 30 02:48:37.715493 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.715428 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:37.876868 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.876828 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7\" (UniqueName: \"kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7\") pod \"e05576d1-f3d8-4966-aab1-50743855e74c\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " Oct 30 02:48:37.877075 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.876893 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume\") pod \"e05576d1-f3d8-4966-aab1-50743855e74c\" (UID: \"e05576d1-f3d8-4966-aab1-50743855e74c\") " Oct 30 02:48:37.877367 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.877337 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume" pod "e05576d1-f3d8-4966-aab1-50743855e74c" (UID: "e05576d1-f3d8-4966-aab1-50743855e74c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:48:37.879197 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.879173 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7" pod "e05576d1-f3d8-4966-aab1-50743855e74c" (UID: "e05576d1-f3d8-4966-aab1-50743855e74c"). InnerVolumeSpecName "kube-api-access-z4cb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:37.978300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.978220 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4cb7\" (UniqueName: \"kubernetes.io/projected/e05576d1-f3d8-4966-aab1-50743855e74c-kube-api-access-z4cb7\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:37.978300 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.978252 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05576d1-f3d8-4966-aab1-50743855e74c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:37.994292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.994259 6275 generic.go:356] "Generic (PLEG): container finished" podID="e05576d1-f3d8-4966-aab1-50743855e74c" containerID="8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548" exitCode=0 Oct 30 02:48:37.994448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.994307 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" Oct 30 02:48:37.994448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.994320 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" event={"ID":"e05576d1-f3d8-4966-aab1-50743855e74c","Type":"ContainerDied","Data":"8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548"} Oct 30 02:48:37.994448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.994363 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk" event={"ID":"e05576d1-f3d8-4966-aab1-50743855e74c","Type":"ContainerDied","Data":"6f68cd4c83a273250a03c4d23eee40cc72bc9110d837462c0697e561702b962b"} Oct 30 02:48:37.994448 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:37.994383 6275 scope.go:122] "RemoveContainer" containerID="8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548" Oct 30 02:48:38.003085 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:38.003068 6275 scope.go:122] "RemoveContainer" containerID="8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548" Oct 30 02:48:38.003415 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:48:38.003382 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548\": container with ID starting with 8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548 not found: ID does not exist" containerID="8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548" Oct 30 02:48:38.003533 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:38.003424 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548"} err="failed to get container status \"8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548\": rpc error: code = NotFound desc = could not find container \"8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548\": container with ID starting with 8bfa6771a2fbcebf69157b685b228119023739a26bf444029910916c4e03e548 not found: ID does not exist" Oct 30 02:48:38.040916 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:38.040874 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk"] Oct 30 02:48:38.051186 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:38.051159 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0944980-7f77d8bd9b-t6tkk"] Oct 30 02:48:39.737282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:39.737250 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05576d1-f3d8-4966-aab1-50743855e74c" path="/var/lib/kubelet/pods/e05576d1-f3d8-4966-aab1-50743855e74c/volumes" Oct 30 02:48:42.143485 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.143443 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75b4575b56-7wgng"] Oct 30 02:48:42.144144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.143706 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75b4575b56-7wgng" podUID="400efcf6-470e-4589-9ad0-fb0c4458116b" containerName="manager" containerID="cri-o://a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9" gracePeriod=30 Oct 30 02:48:42.409633 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.409554 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 02:48:42.521649 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.521609 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs\") pod \"400efcf6-470e-4589-9ad0-fb0c4458116b\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " Oct 30 02:48:42.521849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.521681 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert\") pod \"400efcf6-470e-4589-9ad0-fb0c4458116b\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " Oct 30 02:48:42.521849 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.521839 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5\" (UniqueName: \"kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5\") pod \"400efcf6-470e-4589-9ad0-fb0c4458116b\" (UID: \"400efcf6-470e-4589-9ad0-fb0c4458116b\") " Oct 30 02:48:42.524314 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.524282 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs" pod "400efcf6-470e-4589-9ad0-fb0c4458116b" (UID: "400efcf6-470e-4589-9ad0-fb0c4458116b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:48:42.524419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.524296 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert" pod "400efcf6-470e-4589-9ad0-fb0c4458116b" (UID: "400efcf6-470e-4589-9ad0-fb0c4458116b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:48:42.526232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.526212 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5" pod "400efcf6-470e-4589-9ad0-fb0c4458116b" (UID: "400efcf6-470e-4589-9ad0-fb0c4458116b"). InnerVolumeSpecName "kube-api-access-sv6n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:42.623631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.623596 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:42.623631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.623627 6275 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/400efcf6-470e-4589-9ad0-fb0c4458116b-webhook-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:42.623631 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:42.623637 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6n5\" (UniqueName: \"kubernetes.io/projected/400efcf6-470e-4589-9ad0-fb0c4458116b-kube-api-access-sv6n5\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.020528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.020487 6275 generic.go:356] "Generic (PLEG): container finished" podID="400efcf6-470e-4589-9ad0-fb0c4458116b" containerID="a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9" exitCode=0 Oct 30 02:48:43.020719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.020546 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-7wgng" Oct 30 02:48:43.020719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.020579 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-7wgng" event={"ID":"400efcf6-470e-4589-9ad0-fb0c4458116b","Type":"ContainerDied","Data":"a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9"} Oct 30 02:48:43.020719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.020645 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-7wgng" event={"ID":"400efcf6-470e-4589-9ad0-fb0c4458116b","Type":"ContainerDied","Data":"a3aef0c1ba02c4a74bd62d0b8ddc90374b87ebc98d79937f4c203d5ad6143e08"} Oct 30 02:48:43.020719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.020678 6275 scope.go:122] "RemoveContainer" containerID="a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9" Oct 30 02:48:43.032875 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.032849 6275 scope.go:122] "RemoveContainer" containerID="a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9" Oct 30 02:48:43.033180 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:48:43.033158 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9\": container with ID starting with a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9 not found: ID does not exist" containerID="a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9" Oct 30 02:48:43.033253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.033188 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9"} err="failed to get container status \"a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9\": rpc error: code = NotFound desc = could not find container \"a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9\": container with ID starting with a42fd3c493dbe236327f4107ece109742253f7a4e151e41ee623d4f0fb5fc9a9 not found: ID does not exist" Oct 30 02:48:43.085498 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.085465 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75b4575b56-7wgng"] Oct 30 02:48:43.085748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.085731 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75b4575b56-7wgng"] Oct 30 02:48:43.736639 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:43.736606 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400efcf6-470e-4589-9ad0-fb0c4458116b" path="/var/lib/kubelet/pods/400efcf6-470e-4589-9ad0-fb0c4458116b/volumes" Oct 30 02:48:44.900766 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.900722 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75b4575b56-4865l"] Oct 30 02:48:44.901364 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901343 6275 cpu_manager.go:414] "containerMap: removing container" podUID="62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" containerName="maas-api" Oct 30 02:48:44.901458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901366 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" containerName="maas-api" Oct 30 02:48:44.901458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901390 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e05576d1-f3d8-4966-aab1-50743855e74c" containerName="payload-processing" Oct 30 02:48:44.901458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901398 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05576d1-f3d8-4966-aab1-50743855e74c" containerName="payload-processing" Oct 30 02:48:44.901458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901414 6275 cpu_manager.go:414] "containerMap: removing container" podUID="400efcf6-470e-4589-9ad0-fb0c4458116b" containerName="manager" Oct 30 02:48:44.901458 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901424 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400efcf6-470e-4589-9ad0-fb0c4458116b" containerName="manager" Oct 30 02:48:44.901742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901591 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e4f4ed-03e2-4849-9d0a-4cc9a2caf7a4" containerName="maas-api" Oct 30 02:48:44.901742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901610 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05576d1-f3d8-4966-aab1-50743855e74c" containerName="payload-processing" Oct 30 02:48:44.901742 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.901623 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="400efcf6-470e-4589-9ad0-fb0c4458116b" containerName="manager" Oct 30 02:48:44.905904 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.905881 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:44.919083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.919060 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pm2l8\"" Oct 30 02:48:44.929494 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.929472 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 02:48:44.929624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.929586 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 02:48:44.929719 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.929692 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 02:48:44.931601 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:44.931578 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75b4575b56-4865l"] Oct 30 02:48:45.046421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.046387 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.046609 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.046426 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6fh\" (UniqueName: \"kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.046654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.046616 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.148057 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.148021 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.148282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.148107 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.148282 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.148170 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6fh\" (UniqueName: \"kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.150584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.150546 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.150584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.150573 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.169981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.169924 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6fh\" (UniqueName: \"kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh\") pod \"maas-controller-75b4575b56-4865l\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.219239 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.219214 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:45.519674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:45.519629 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75b4575b56-4865l"] Oct 30 02:48:45.523777 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:48:45.523750 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f874da_9914_400c_985b_3ab49bc98c5c.slice/crio-71040e65eb8c25f9f3ebd1522d8a6be73b2eb5bba0d9fcbf3010dd61c67c0522 WatchSource:0}: Error finding container 71040e65eb8c25f9f3ebd1522d8a6be73b2eb5bba0d9fcbf3010dd61c67c0522: Status 404 returned error can't find the container with id 71040e65eb8c25f9f3ebd1522d8a6be73b2eb5bba0d9fcbf3010dd61c67c0522 Oct 30 02:48:46.036902 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:46.036861 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-4865l" event={"ID":"23f874da-9914-400c-985b-3ab49bc98c5c","Type":"ContainerStarted","Data":"88d421c12528446e53fa7b5faba0579ce543e6b93a04bae0c49471c76a74e51e"} Oct 30 02:48:46.037384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:46.036945 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-4865l" event={"ID":"23f874da-9914-400c-985b-3ab49bc98c5c","Type":"ContainerStarted","Data":"71040e65eb8c25f9f3ebd1522d8a6be73b2eb5bba0d9fcbf3010dd61c67c0522"} Oct 30 02:48:46.037384 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:46.036967 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:48:46.086958 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:46.086877 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75b4575b56-4865l" podStartSLOduration=1.682196166 podStartE2EDuration="2.086860535s" totalImagesPullingTime="404.664369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:46.076259998 +0000 UTC m=+4754.468067632" watchObservedRunningTime="2026-10-30 02:48:46.086860535 +0000 UTC m=+4754.478668167" Oct 30 02:48:46.697006 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:48:46.696968 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:48:57.044114 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:48:57.044083 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:49:06.132731 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.132682 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt"] Oct 30 02:49:06.136358 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.136340 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.162172 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.162139 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt"] Oct 30 02:49:06.251177 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251113 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251221 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbnq\" (UniqueName: \"kubernetes.io/projected/b4fb616c-0b22-482d-86e1-dade8fda8955-kube-api-access-4pbnq\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251265 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fb616c-0b22-482d-86e1-dade8fda8955-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251301 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251340 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251360 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251359 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251387 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251487 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.251556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.251535 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352289 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352252 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352305 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352351 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352374 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbnq\" (UniqueName: \"kubernetes.io/projected/b4fb616c-0b22-482d-86e1-dade8fda8955-kube-api-access-4pbnq\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352393 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fb616c-0b22-482d-86e1-dade8fda8955-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352480 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352420 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352571 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352620 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352667 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.352790 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352713 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.353008 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.352805 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.353065 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.353033 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.353169 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.353096 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.353323 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.353300 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fb616c-0b22-482d-86e1-dade8fda8955-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.354769 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.354744 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.355050 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.355028 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.373816 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.373791 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fb616c-0b22-482d-86e1-dade8fda8955-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.374052 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.374032 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbnq\" (UniqueName: \"kubernetes.io/projected/b4fb616c-0b22-482d-86e1-dade8fda8955-kube-api-access-4pbnq\") pod \"maas-default-gateway-openshift-default-75d7c785f-lxtlt\" (UID: \"b4fb616c-0b22-482d-86e1-dade8fda8955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.451998 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.451918 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:06.772412 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.772367 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt"] Oct 30 02:49:06.776180 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:49:06.776151 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fb616c_0b22_482d_86e1_dade8fda8955.slice/crio-08baef17df2ec77c6df63e20d920ee811c4587b4ba91be850eed91a3575a81c2 WatchSource:0}: Error finding container 08baef17df2ec77c6df63e20d920ee811c4587b4ba91be850eed91a3575a81c2: Status 404 returned error can't find the container with id 08baef17df2ec77c6df63e20d920ee811c4587b4ba91be850eed91a3575a81c2 Oct 30 02:49:06.778366 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.778331 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:49:06.778440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.778403 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:49:06.778440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:06.778433 6275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 02:49:07.135307 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:07.135219 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" event={"ID":"b4fb616c-0b22-482d-86e1-dade8fda8955","Type":"ContainerStarted","Data":"d3dabcc50e6ebf7080849940e6ce6383d1b5b319f1f55f226b5b527480d747aa"} Oct 30 02:49:07.135683 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:07.135327 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" event={"ID":"b4fb616c-0b22-482d-86e1-dade8fda8955","Type":"ContainerStarted","Data":"08baef17df2ec77c6df63e20d920ee811c4587b4ba91be850eed91a3575a81c2"} Oct 30 02:49:07.181073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:07.181009 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podStartSLOduration=1.180991763 podStartE2EDuration="1.180991763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:07.178685935 +0000 UTC m=+4775.570493568" watchObservedRunningTime="2026-10-30 02:49:07.180991763 +0000 UTC m=+4775.572799398" Oct 30 02:49:07.459324 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:07.459226 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:07.459676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:07.459651 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:07.459847 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:07.459822 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:08.455624 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:08.455581 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:08.456306 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:08.456268 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:09.454004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:09.453531 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:09.454004 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:09.453614 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:10.453526 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:10.453490 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:10.454155 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:10.454103 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:11.463544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:11.463501 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:11.464090 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:11.463581 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:12.457203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:12.454488 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:12.457203 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:12.454552 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:13.452909 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:13.452871 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Oct 30 02:49:13.453454 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:13.452927 6275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Oct 30 02:49:15.014675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.014631 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75b4575b56-4865l"] Oct 30 02:49:15.015117 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.014892 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75b4575b56-4865l" podUID="23f874da-9914-400c-985b-3ab49bc98c5c" containerName="manager" containerID="cri-o://88d421c12528446e53fa7b5faba0579ce543e6b93a04bae0c49471c76a74e51e" gracePeriod=30 Oct 30 02:49:15.079723 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.079683 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:15.080079 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.080057 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:15.080924 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.080904 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" Oct 30 02:49:15.203291 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.203259 6275 generic.go:356] "Generic (PLEG): container finished" podID="23f874da-9914-400c-985b-3ab49bc98c5c" containerID="88d421c12528446e53fa7b5faba0579ce543e6b93a04bae0c49471c76a74e51e" exitCode=0 Oct 30 02:49:15.203460 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.203336 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-4865l" event={"ID":"23f874da-9914-400c-985b-3ab49bc98c5c","Type":"ContainerDied","Data":"88d421c12528446e53fa7b5faba0579ce543e6b93a04bae0c49471c76a74e51e"} Oct 30 02:49:15.303966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.303938 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:49:15.451285 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.451254 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh\" (UniqueName: \"kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh\") pod \"23f874da-9914-400c-985b-3ab49bc98c5c\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " Oct 30 02:49:15.451466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.451389 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs\") pod \"23f874da-9914-400c-985b-3ab49bc98c5c\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " Oct 30 02:49:15.451466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.451413 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert\") pod \"23f874da-9914-400c-985b-3ab49bc98c5c\" (UID: \"23f874da-9914-400c-985b-3ab49bc98c5c\") " Oct 30 02:49:15.453405 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.453366 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh" pod "23f874da-9914-400c-985b-3ab49bc98c5c" (UID: "23f874da-9914-400c-985b-3ab49bc98c5c"). InnerVolumeSpecName "kube-api-access-kt6fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:49:15.453405 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.453391 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert" pod "23f874da-9914-400c-985b-3ab49bc98c5c" (UID: "23f874da-9914-400c-985b-3ab49bc98c5c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:49:15.453606 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.453407 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs" pod "23f874da-9914-400c-985b-3ab49bc98c5c" (UID: "23f874da-9914-400c-985b-3ab49bc98c5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:49:15.552759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.552663 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:49:15.552759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.552702 6275 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23f874da-9914-400c-985b-3ab49bc98c5c-webhook-cert\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:49:15.552759 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:15.552719 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt6fh\" (UniqueName: \"kubernetes.io/projected/23f874da-9914-400c-985b-3ab49bc98c5c-kube-api-access-kt6fh\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:49:16.209889 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:16.209859 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-4865l" Oct 30 02:49:16.210355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:16.209894 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-4865l" event={"ID":"23f874da-9914-400c-985b-3ab49bc98c5c","Type":"ContainerDied","Data":"71040e65eb8c25f9f3ebd1522d8a6be73b2eb5bba0d9fcbf3010dd61c67c0522"} Oct 30 02:49:16.210355 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:16.209957 6275 scope.go:122] "RemoveContainer" containerID="88d421c12528446e53fa7b5faba0579ce543e6b93a04bae0c49471c76a74e51e" Oct 30 02:49:16.258021 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:16.257994 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75b4575b56-4865l"] Oct 30 02:49:16.268672 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:16.268643 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75b4575b56-4865l"] Oct 30 02:49:16.695770 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:49:16.695736 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:49:17.467346 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.467305 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75b4575b56-vfzpw"] Oct 30 02:49:17.467914 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.467892 6275 cpu_manager.go:414] "containerMap: removing container" podUID="23f874da-9914-400c-985b-3ab49bc98c5c" containerName="manager" Oct 30 02:49:17.468022 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.467916 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f874da-9914-400c-985b-3ab49bc98c5c" containerName="manager" Oct 30 02:49:17.468081 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.468024 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f874da-9914-400c-985b-3ab49bc98c5c" containerName="manager" Oct 30 02:49:17.472728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.472705 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.496438 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.496404 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 02:49:17.496581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.496496 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 02:49:17.496647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.496602 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 02:49:17.496647 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.496612 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pm2l8\"" Oct 30 02:49:17.498390 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.498369 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75b4575b56-vfzpw"] Oct 30 02:49:17.572337 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.572289 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3452c78-90c4-4234-a319-9de51f7d97e0-webhook-cert\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.572506 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.572485 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfpv\" (UniqueName: \"kubernetes.io/projected/f3452c78-90c4-4234-a319-9de51f7d97e0-kube-api-access-8gfpv\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.572571 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.572522 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3452c78-90c4-4234-a319-9de51f7d97e0-metrics-certs\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.673789 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.673751 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3452c78-90c4-4234-a319-9de51f7d97e0-webhook-cert\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.673972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.673875 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfpv\" (UniqueName: \"kubernetes.io/projected/f3452c78-90c4-4234-a319-9de51f7d97e0-kube-api-access-8gfpv\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.673972 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.673901 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3452c78-90c4-4234-a319-9de51f7d97e0-metrics-certs\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.676304 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.676281 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3452c78-90c4-4234-a319-9de51f7d97e0-metrics-certs\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.676440 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.676403 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3452c78-90c4-4234-a319-9de51f7d97e0-webhook-cert\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.695470 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.695448 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfpv\" (UniqueName: \"kubernetes.io/projected/f3452c78-90c4-4234-a319-9de51f7d97e0-kube-api-access-8gfpv\") pod \"maas-controller-75b4575b56-vfzpw\" (UID: \"f3452c78-90c4-4234-a319-9de51f7d97e0\") " pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:17.737876 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.737814 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f874da-9914-400c-985b-3ab49bc98c5c" path="/var/lib/kubelet/pods/23f874da-9914-400c-985b-3ab49bc98c5c/volumes" Oct 30 02:49:17.786068 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:17.786043 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:18.091343 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:18.091309 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75b4575b56-vfzpw"] Oct 30 02:49:18.095423 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:49:18.095395 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3452c78_90c4_4234_a319_9de51f7d97e0.slice/crio-f49674ad040839814016b83c6704b41810a5cd94a60fa4023a8e4946c3a45ecf WatchSource:0}: Error finding container f49674ad040839814016b83c6704b41810a5cd94a60fa4023a8e4946c3a45ecf: Status 404 returned error can't find the container with id f49674ad040839814016b83c6704b41810a5cd94a60fa4023a8e4946c3a45ecf Oct 30 02:49:18.223170 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:18.223109 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-vfzpw" event={"ID":"f3452c78-90c4-4234-a319-9de51f7d97e0","Type":"ContainerStarted","Data":"f49674ad040839814016b83c6704b41810a5cd94a60fa4023a8e4946c3a45ecf"} Oct 30 02:49:19.231819 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:19.231781 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75b4575b56-vfzpw" event={"ID":"f3452c78-90c4-4234-a319-9de51f7d97e0","Type":"ContainerStarted","Data":"95b16319be161aa12390d3d9bacbc1dff21d69b1ee2f87ce406bd4aa436b515d"} Oct 30 02:49:19.232285 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:19.231847 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:19.273032 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:19.272979 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75b4575b56-vfzpw" podStartSLOduration=1.9431369410000001 podStartE2EDuration="2.27296328s" totalImagesPullingTime="329.826339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:19.272082783 +0000 UTC m=+4787.663890416" watchObservedRunningTime="2026-10-30 02:49:19.27296328 +0000 UTC m=+4787.664770915" Oct 30 02:49:30.239219 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:30.239179 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75b4575b56-vfzpw" Oct 30 02:49:32.709230 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:32.709204 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:49:32.721411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:49:32.721387 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:49:46.696424 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:49:46.696387 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:50:11.030451 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:11.030413 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx"] Oct 30 02:50:11.040707 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:11.040681 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab41952-jdjkx"] Oct 30 02:50:11.737292 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:11.737248 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82f90c0-f1a2-46b7-8dd5-117b35481b71" path="/var/lib/kubelet/pods/e82f90c0-f1a2-46b7-8dd5-117b35481b71/volumes" Oct 30 02:50:16.696058 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:50:16.696020 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:50:46.737737 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:50:46.737651 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:50:49.953102 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:49.953059 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6"] Oct 30 02:50:49.956864 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:49.956834 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:49.975303 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:49.975277 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-8b3c5a4d-epp-sa-dockercfg-57rbw\"" Oct 30 02:50:49.985646 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:49.985620 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-8b3c5a4d-kserve-self-signed-certs\"" Oct 30 02:50:49.987584 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:49.987558 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6"] Oct 30 02:50:50.133544 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.133505 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs\") pod \"e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.133748 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.133646 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwmn\" (UniqueName: \"kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn\") pod \"e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.234368 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.234267 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs\") pod \"e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.234556 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.234377 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwmn\" (UniqueName: \"kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn\") pod \"e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.243800 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.240493 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs\") pod \"e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.256718 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.256692 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwmn\" (UniqueName: \"kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn\") pod \"e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.270863 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.270830 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:50.570645 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.570599 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6"] Oct 30 02:50:50.574531 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:50:50.574504 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50980c1_3a2b_4e7c_b025_a6ffc56f2282.slice/crio-391ac9b74bdf5c1e17b1a0cf3f7f6648dd99b20948a4a66a3200a978e2aed84b WatchSource:0}: Error finding container 391ac9b74bdf5c1e17b1a0cf3f7f6648dd99b20948a4a66a3200a978e2aed84b: Status 404 returned error can't find the container with id 391ac9b74bdf5c1e17b1a0cf3f7f6648dd99b20948a4a66a3200a978e2aed84b Oct 30 02:50:50.734011 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:50.733979 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" event={"ID":"a50980c1-3a2b-4e7c-b025-a6ffc56f2282","Type":"ContainerStarted","Data":"391ac9b74bdf5c1e17b1a0cf3f7f6648dd99b20948a4a66a3200a978e2aed84b"} Oct 30 02:50:52.748209 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:52.748167 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" event={"ID":"a50980c1-3a2b-4e7c-b025-a6ffc56f2282","Type":"ContainerStarted","Data":"4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297"} Oct 30 02:50:52.748751 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:52.748292 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:50:52.790083 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:50:52.790017 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podStartSLOduration=2.165190173 podStartE2EDuration="3.789999185s" totalImagesPullingTime="1.624809012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:52.787428028 +0000 UTC m=+4881.179235663" watchObservedRunningTime="2026-10-30 02:50:52.789999185 +0000 UTC m=+4881.181806819" Oct 30 02:51:16.695666 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:51:16.695630 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:51:23.764777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:51:23.764750 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:51:36.285183 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:51:36.285152 6275 scope.go:122] "RemoveContainer" containerID="9cfc6cb4bd68dac18804f2a98637a975afd7944a6540f72d8336c7a5d9a81007" Oct 30 02:51:46.695054 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:51:46.695016 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:51:49.617796 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:51:49.617754 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6"] Oct 30 02:51:53.765092 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:51:53.765048 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:52:03.765487 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:03.765448 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:52:04.646643 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:04.646581 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" containerID="cri-o://4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297" gracePeriod=45 Oct 30 02:52:13.767365 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:13.767265 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:52:13.768276 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:13.768233 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:52:16.696190 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:52:16.696150 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:52:23.764955 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:23.764909 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:52:33.765673 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:33.765634 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:52:34.910284 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:34.910228 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:52:35.055109 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.055074 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs\" (UniqueName: \"kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs\") pod \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " Oct 30 02:52:35.055296 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.055223 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn\" (UniqueName: \"kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn\") pod \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\" (UID: \"a50980c1-3a2b-4e7c-b025-a6ffc56f2282\") " Oct 30 02:52:35.057378 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.057333 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs" pod "a50980c1-3a2b-4e7c-b025-a6ffc56f2282" (UID: "a50980c1-3a2b-4e7c-b025-a6ffc56f2282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:52:35.057540 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.057382 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn" pod "a50980c1-3a2b-4e7c-b025-a6ffc56f2282" (UID: "a50980c1-3a2b-4e7c-b025-a6ffc56f2282"). InnerVolumeSpecName "kube-api-access-xkwmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:52:35.156845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.156818 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:52:35.156845 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.156841 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwmn\" (UniqueName: \"kubernetes.io/projected/a50980c1-3a2b-4e7c-b025-a6ffc56f2282-kube-api-access-xkwmn\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:52:35.356356 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.356324 6275 generic.go:356] "Generic (PLEG): container finished" podID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerID="4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297" exitCode=0 Oct 30 02:52:35.356548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.356373 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" Oct 30 02:52:35.356548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.356414 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" event={"ID":"a50980c1-3a2b-4e7c-b025-a6ffc56f2282","Type":"ContainerDied","Data":"4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297"} Oct 30 02:52:35.356548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.356465 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6" event={"ID":"a50980c1-3a2b-4e7c-b025-a6ffc56f2282","Type":"ContainerDied","Data":"391ac9b74bdf5c1e17b1a0cf3f7f6648dd99b20948a4a66a3200a978e2aed84b"} Oct 30 02:52:35.356548 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.356486 6275 scope.go:122] "RemoveContainer" containerID="4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297" Oct 30 02:52:35.366025 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.366004 6275 scope.go:122] "RemoveContainer" containerID="4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297" Oct 30 02:52:35.366316 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:52:35.366296 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297\": container with ID starting with 4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297 not found: ID does not exist" containerID="4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297" Oct 30 02:52:35.366400 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.366324 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297"} err="failed to get container status \"4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297\": rpc error: code = NotFound desc = could not find container \"4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297\": container with ID starting with 4e8c162384a7f4a664397861b2f39785a4ed732c192bd1c16acb4f729d149297 not found: ID does not exist" Oct 30 02:52:35.401934 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.401904 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6"] Oct 30 02:52:35.406163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.406143 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-router-scheduler-65c46dcd44-vrsq6"] Oct 30 02:52:35.737528 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:52:35.737438 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" path="/var/lib/kubelet/pods/a50980c1-3a2b-4e7c-b025-a6ffc56f2282/volumes" Oct 30 02:52:46.696029 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:52:46.695988 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:53:16.696413 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:53:16.696374 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:53:27.132162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.132105 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn"] Oct 30 02:53:27.132966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.132771 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" Oct 30 02:53:27.132966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.132791 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" Oct 30 02:53:27.132966 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.132912 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50980c1-3a2b-4e7c-b025-a6ffc56f2282" containerName="main" Oct 30 02:53:27.135960 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.135941 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.160763 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.160735 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-50b514ff-kserve-self-signed-certs\"" Oct 30 02:53:27.162754 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.162727 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn"] Oct 30 02:53:27.232106 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.232064 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.232348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.232250 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.232348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.232313 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmh8\" (UniqueName: \"kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.232476 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.232451 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.232529 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.232515 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.232593 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.232568 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.334674 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.334626 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.334879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.334706 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.334879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.334734 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.334879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.334779 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.334879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.334839 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.334879 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.334870 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmh8\" (UniqueName: \"kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.335427 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.335402 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.335676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.335653 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.335833 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.335808 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.337312 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.337284 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.337597 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.337579 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.356959 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.356932 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmh8\" (UniqueName: \"kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8\") pod \"e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.447728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.447647 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:27.748724 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:27.748617 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn"] Oct 30 02:53:27.753293 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:53:27.753263 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda93c192a_e293_43d7_907f_9b200072ee00.slice/crio-f323ed5d03e0eadf983cfdeb07cceb2edf864b45a6e4f818ef9f764ad572765e WatchSource:0}: Error finding container f323ed5d03e0eadf983cfdeb07cceb2edf864b45a6e4f818ef9f764ad572765e: Status 404 returned error can't find the container with id f323ed5d03e0eadf983cfdeb07cceb2edf864b45a6e4f818ef9f764ad572765e Oct 30 02:53:28.620077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:28.620035 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" event={"ID":"a93c192a-e293-43d7-907f-9b200072ee00","Type":"ContainerStarted","Data":"d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651"} Oct 30 02:53:28.620077 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:28.620083 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" event={"ID":"a93c192a-e293-43d7-907f-9b200072ee00","Type":"ContainerStarted","Data":"f323ed5d03e0eadf983cfdeb07cceb2edf864b45a6e4f818ef9f764ad572765e"} Oct 30 02:53:28.661620 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:28.661560 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" podStartSLOduration=1.6615473299999999 podStartE2EDuration="1.66154733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:28.660357559 +0000 UTC m=+5037.052165193" watchObservedRunningTime="2026-10-30 02:53:28.66154733 +0000 UTC m=+5037.053354960" Oct 30 02:53:29.448100 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:29.448059 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:37.448145 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:37.448084 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:37.460715 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:37.460679 6275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:37.682455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:37.682418 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:53:46.731957 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:53:46.731845 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:53:50.097261 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:50.097215 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:53:50.097877 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:53:50.097846 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:54:16.696360 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:54:16.696319 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:54:17.275554 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:17.275505 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn"] Oct 30 02:54:32.296809 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.296737 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" podUID="a93c192a-e293-43d7-907f-9b200072ee00" containerName="main" containerID="cri-o://d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651" gracePeriod=45 Oct 30 02:54:32.553006 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.552946 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:54:32.624283 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624255 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm\") pod \"a93c192a-e293-43d7-907f-9b200072ee00\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " Oct 30 02:54:32.624419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624303 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home\") pod \"a93c192a-e293-43d7-907f-9b200072ee00\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " Oct 30 02:54:32.624419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624334 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8\" (UniqueName: \"kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8\") pod \"a93c192a-e293-43d7-907f-9b200072ee00\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " Oct 30 02:54:32.624419 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624388 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs\" (UniqueName: \"kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs\") pod \"a93c192a-e293-43d7-907f-9b200072ee00\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " Oct 30 02:54:32.624605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624440 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache\") pod \"a93c192a-e293-43d7-907f-9b200072ee00\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " Oct 30 02:54:32.624605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624485 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir\") pod \"a93c192a-e293-43d7-907f-9b200072ee00\" (UID: \"a93c192a-e293-43d7-907f-9b200072ee00\") " Oct 30 02:54:32.624605 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624568 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home" pod "a93c192a-e293-43d7-907f-9b200072ee00" (UID: "a93c192a-e293-43d7-907f-9b200072ee00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:32.624778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624745 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache" pod "a93c192a-e293-43d7-907f-9b200072ee00" (UID: "a93c192a-e293-43d7-907f-9b200072ee00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:32.624778 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624752 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir" pod "a93c192a-e293-43d7-907f-9b200072ee00" (UID: "a93c192a-e293-43d7-907f-9b200072ee00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:32.624930 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624912 6275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-home\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.624978 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624937 6275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-model-cache\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.625027 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.624976 6275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-tmp-dir\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.626349 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.626329 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm" pod "a93c192a-e293-43d7-907f-9b200072ee00" (UID: "a93c192a-e293-43d7-907f-9b200072ee00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:32.626442 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.626424 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs" pod "a93c192a-e293-43d7-907f-9b200072ee00" (UID: "a93c192a-e293-43d7-907f-9b200072ee00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:54:32.626590 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.626569 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8" pod "a93c192a-e293-43d7-907f-9b200072ee00" (UID: "a93c192a-e293-43d7-907f-9b200072ee00"). InnerVolumeSpecName "kube-api-access-bzmh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:54:32.725622 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.725592 6275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93c192a-e293-43d7-907f-9b200072ee00-dshm\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.725937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.725623 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmh8\" (UniqueName: \"kubernetes.io/projected/a93c192a-e293-43d7-907f-9b200072ee00-kube-api-access-bzmh8\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.725937 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.725635 6275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93c192a-e293-43d7-907f-9b200072ee00-tls-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.738887 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.738866 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:54:32.751728 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:32.751711 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:54:33.022591 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.022551 6275 generic.go:356] "Generic (PLEG): container finished" podID="a93c192a-e293-43d7-907f-9b200072ee00" containerID="d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651" exitCode=0 Oct 30 02:54:33.022771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.022602 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" Oct 30 02:54:33.022771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.022606 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" event={"ID":"a93c192a-e293-43d7-907f-9b200072ee00","Type":"ContainerDied","Data":"d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651"} Oct 30 02:54:33.022771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.022705 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn" event={"ID":"a93c192a-e293-43d7-907f-9b200072ee00","Type":"ContainerDied","Data":"f323ed5d03e0eadf983cfdeb07cceb2edf864b45a6e4f818ef9f764ad572765e"} Oct 30 02:54:33.022771 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.022723 6275 scope.go:122] "RemoveContainer" containerID="d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651" Oct 30 02:54:33.032163 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.032150 6275 scope.go:122] "RemoveContainer" containerID="d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651" Oct 30 02:54:33.032432 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:54:33.032411 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651\": container with ID starting with d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651 not found: ID does not exist" containerID="d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651" Oct 30 02:54:33.032486 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.032441 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651"} err="failed to get container status \"d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651\": rpc error: code = NotFound desc = could not find container \"d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651\": container with ID starting with d3dda789d2b7058d986390c3eb0c30ae9127040552e9f2968e065866fce3e651 not found: ID does not exist" Oct 30 02:54:33.069456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.069434 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn"] Oct 30 02:54:33.080335 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.080309 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-50b514ff-kserve-5cbd7cb68f-bkfsn"] Oct 30 02:54:33.737223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:33.737193 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93c192a-e293-43d7-907f-9b200072ee00" path="/var/lib/kubelet/pods/a93c192a-e293-43d7-907f-9b200072ee00/volumes" Oct 30 02:54:46.696726 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:54:46.696688 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:54:59.310962 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.310926 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24"] Oct 30 02:54:59.311546 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.311392 6275 cpu_manager.go:414] "containerMap: removing container" podUID="a93c192a-e293-43d7-907f-9b200072ee00" containerName="main" Oct 30 02:54:59.311546 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.311405 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93c192a-e293-43d7-907f-9b200072ee00" containerName="main" Oct 30 02:54:59.311546 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.311482 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93c192a-e293-43d7-907f-9b200072ee00" containerName="main" Oct 30 02:54:59.314699 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.314676 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.335226 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.335197 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24"] Oct 30 02:54:59.435785 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.435745 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume\") pod \"payload-pre-processing-8dd7b7cc5-ngv24\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.435984 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.435866 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xk6\" (UniqueName: \"kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6\") pod \"payload-pre-processing-8dd7b7cc5-ngv24\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.537210 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.537169 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume\") pod \"payload-pre-processing-8dd7b7cc5-ngv24\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.537399 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.537260 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xk6\" (UniqueName: \"kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6\") pod \"payload-pre-processing-8dd7b7cc5-ngv24\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.537903 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.537880 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume\") pod \"payload-pre-processing-8dd7b7cc5-ngv24\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.559097 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.559069 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xk6\" (UniqueName: \"kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6\") pod \"payload-pre-processing-8dd7b7cc5-ngv24\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.628425 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.628336 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:54:59.931393 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:54:59.931363 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24"] Oct 30 02:54:59.933582 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:54:59.933552 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f14e0d_dd87_456a_88b2_c2c9a762403c.slice/crio-481b3dddf6484c1f98a47ab294bfff83029281a8989f476dd8d2d0760197e1b0 WatchSource:0}: Error finding container 481b3dddf6484c1f98a47ab294bfff83029281a8989f476dd8d2d0760197e1b0: Status 404 returned error can't find the container with id 481b3dddf6484c1f98a47ab294bfff83029281a8989f476dd8d2d0760197e1b0 Oct 30 02:55:00.205561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:00.205451 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" event={"ID":"d7f14e0d-dd87-456a-88b2-c2c9a762403c","Type":"ContainerStarted","Data":"f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9"} Oct 30 02:55:00.205561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:00.205506 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" event={"ID":"d7f14e0d-dd87-456a-88b2-c2c9a762403c","Type":"ContainerStarted","Data":"481b3dddf6484c1f98a47ab294bfff83029281a8989f476dd8d2d0760197e1b0"} Oct 30 02:55:00.205561 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:00.205554 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:55:00.243826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:00.243771 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" podStartSLOduration=1.243758365 podStartE2EDuration="1.243758365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:00.243383856 +0000 UTC m=+5128.635191490" watchObservedRunningTime="2026-10-30 02:55:00.243758365 +0000 UTC m=+5128.635565997" Oct 30 02:55:11.213376 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:11.213345 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:55:11.281901 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:11.281864 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx"] Oct 30 02:55:16.283265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.283153 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" podUID="e77bd722-7979-4f70-a35b-e131590ff267" containerName="payload-pre-processing" containerID="cri-o://fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228" gracePeriod=25 Oct 30 02:55:16.527567 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.527541 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 02:55:16.623034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.622938 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4\" (UniqueName: \"kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4\") pod \"e77bd722-7979-4f70-a35b-e131590ff267\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " Oct 30 02:55:16.623034 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.623005 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume\") pod \"e77bd722-7979-4f70-a35b-e131590ff267\" (UID: \"e77bd722-7979-4f70-a35b-e131590ff267\") " Oct 30 02:55:16.623421 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.623400 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume" pod "e77bd722-7979-4f70-a35b-e131590ff267" (UID: "e77bd722-7979-4f70-a35b-e131590ff267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:55:16.625265 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.625241 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4" pod "e77bd722-7979-4f70-a35b-e131590ff267" (UID: "e77bd722-7979-4f70-a35b-e131590ff267"). InnerVolumeSpecName "kube-api-access-84xh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:55:16.698590 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:55:16.698555 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:55:16.724061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.724023 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77bd722-7979-4f70-a35b-e131590ff267-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:55:16.724061 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:16.724056 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xh4\" (UniqueName: \"kubernetes.io/projected/e77bd722-7979-4f70-a35b-e131590ff267-kube-api-access-84xh4\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:55:17.287039 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.287005 6275 generic.go:356] "Generic (PLEG): container finished" podID="e77bd722-7979-4f70-a35b-e131590ff267" containerID="fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228" exitCode=0 Oct 30 02:55:17.287520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.287057 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" Oct 30 02:55:17.287520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.287096 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" event={"ID":"e77bd722-7979-4f70-a35b-e131590ff267","Type":"ContainerDied","Data":"fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228"} Oct 30 02:55:17.287520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.287177 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx" event={"ID":"e77bd722-7979-4f70-a35b-e131590ff267","Type":"ContainerDied","Data":"c24ab42c0d03f0f99b0500969c3c4701c20968a5309ac8bb1c5e2ac61f13d9ae"} Oct 30 02:55:17.287520 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.287201 6275 scope.go:122] "RemoveContainer" containerID="fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228" Oct 30 02:55:17.297100 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.297073 6275 scope.go:122] "RemoveContainer" containerID="fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228" Oct 30 02:55:17.297397 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:55:17.297377 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228\": container with ID starting with fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228 not found: ID does not exist" containerID="fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228" Oct 30 02:55:17.297453 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.297406 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228"} err="failed to get container status \"fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228\": rpc error: code = NotFound desc = could not find container \"fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228\": container with ID starting with fd72ba52d3142678c527e2074920a3d6de8f96d19ccd8a40e8c81e557baac228 not found: ID does not exist" Oct 30 02:55:17.333891 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.333835 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx"] Oct 30 02:55:17.343981 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.343957 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xsvkx"] Oct 30 02:55:17.738581 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:17.738501 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77bd722-7979-4f70-a35b-e131590ff267" path="/var/lib/kubelet/pods/e77bd722-7979-4f70-a35b-e131590ff267/volumes" Oct 30 02:55:25.730988 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:25.730953 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-qfc9q"] Oct 30 02:55:27.601053 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.601014 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-75c985bd9-6thmd"] Oct 30 02:55:27.601676 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.601654 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e77bd722-7979-4f70-a35b-e131590ff267" containerName="payload-pre-processing" Oct 30 02:55:27.601738 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.601679 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77bd722-7979-4f70-a35b-e131590ff267" containerName="payload-pre-processing" Oct 30 02:55:27.601779 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.601769 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77bd722-7979-4f70-a35b-e131590ff267" containerName="payload-pre-processing" Oct 30 02:55:27.606162 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.606116 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.624654 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.624626 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-75c985bd9-6thmd"] Oct 30 02:55:27.730570 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.730531 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9xx\" (UniqueName: \"kubernetes.io/projected/e2362a6d-db63-4753-b0be-625d9d0be0c3-kube-api-access-5h9xx\") pod \"payload-processing-75c985bd9-6thmd\" (UID: \"e2362a6d-db63-4753-b0be-625d9d0be0c3\") " pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.730784 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.730598 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2362a6d-db63-4753-b0be-625d9d0be0c3-plugins-config-volume\") pod \"payload-processing-75c985bd9-6thmd\" (UID: \"e2362a6d-db63-4753-b0be-625d9d0be0c3\") " pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.832201 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.832154 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9xx\" (UniqueName: \"kubernetes.io/projected/e2362a6d-db63-4753-b0be-625d9d0be0c3-kube-api-access-5h9xx\") pod \"payload-processing-75c985bd9-6thmd\" (UID: \"e2362a6d-db63-4753-b0be-625d9d0be0c3\") " pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.832411 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.832243 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2362a6d-db63-4753-b0be-625d9d0be0c3-plugins-config-volume\") pod \"payload-processing-75c985bd9-6thmd\" (UID: \"e2362a6d-db63-4753-b0be-625d9d0be0c3\") " pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.833281 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.833251 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2362a6d-db63-4753-b0be-625d9d0be0c3-plugins-config-volume\") pod \"payload-processing-75c985bd9-6thmd\" (UID: \"e2362a6d-db63-4753-b0be-625d9d0be0c3\") " pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.854422 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.854354 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9xx\" (UniqueName: \"kubernetes.io/projected/e2362a6d-db63-4753-b0be-625d9d0be0c3-kube-api-access-5h9xx\") pod \"payload-processing-75c985bd9-6thmd\" (UID: \"e2362a6d-db63-4753-b0be-625d9d0be0c3\") " pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:27.918629 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:27.918590 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:28.217455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:28.217355 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-75c985bd9-6thmd"] Oct 30 02:55:28.221754 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:55:28.221725 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2362a6d_db63_4753_b0be_625d9d0be0c3.slice/crio-a2e6267ae54c87d2b175e833f640eb75ca50de3bb3ba793986efa12b7c4c4b4a WatchSource:0}: Error finding container a2e6267ae54c87d2b175e833f640eb75ca50de3bb3ba793986efa12b7c4c4b4a: Status 404 returned error can't find the container with id a2e6267ae54c87d2b175e833f640eb75ca50de3bb3ba793986efa12b7c4c4b4a Oct 30 02:55:28.346572 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:28.346538 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" event={"ID":"e2362a6d-db63-4753-b0be-625d9d0be0c3","Type":"ContainerStarted","Data":"00a2f8f6a770d640964ce7ea63e1998bb6e6faed087aa8d5d4f9a8e53ee6a3d3"} Oct 30 02:55:28.346762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:28.346586 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" event={"ID":"e2362a6d-db63-4753-b0be-625d9d0be0c3","Type":"ContainerStarted","Data":"a2e6267ae54c87d2b175e833f640eb75ca50de3bb3ba793986efa12b7c4c4b4a"} Oct 30 02:55:28.346762 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:28.346648 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:28.385651 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:28.385587 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" podStartSLOduration=1.385570977 podStartE2EDuration="1.385570977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:28.384319419 +0000 UTC m=+5156.776127052" watchObservedRunningTime="2026-10-30 02:55:28.385570977 +0000 UTC m=+5156.777378621" Oct 30 02:55:30.731455 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:30.731407 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" podUID="e75c1729-dbe8-4486-a74d-fa6611610662" containerName="payload-processing" containerID="cri-o://026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c" gracePeriod=25 Oct 30 02:55:30.988165 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:30.988100 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 02:55:31.065015 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.064982 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42\" (UniqueName: \"kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42\") pod \"e75c1729-dbe8-4486-a74d-fa6611610662\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " Oct 30 02:55:31.065348 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.065077 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume\") pod \"e75c1729-dbe8-4486-a74d-fa6611610662\" (UID: \"e75c1729-dbe8-4486-a74d-fa6611610662\") " Oct 30 02:55:31.065457 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.065436 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume" pod "e75c1729-dbe8-4486-a74d-fa6611610662" (UID: "e75c1729-dbe8-4486-a74d-fa6611610662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:55:31.067249 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.067220 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42" pod "e75c1729-dbe8-4486-a74d-fa6611610662" (UID: "e75c1729-dbe8-4486-a74d-fa6611610662"). InnerVolumeSpecName "kube-api-access-nfx42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:55:31.166067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.166029 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75c1729-dbe8-4486-a74d-fa6611610662-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:55:31.166067 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.166062 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfx42\" (UniqueName: \"kubernetes.io/projected/e75c1729-dbe8-4486-a74d-fa6611610662-kube-api-access-nfx42\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:55:31.361657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.361620 6275 generic.go:356] "Generic (PLEG): container finished" podID="e75c1729-dbe8-4486-a74d-fa6611610662" containerID="026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c" exitCode=0 Oct 30 02:55:31.361827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.361669 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" Oct 30 02:55:31.361827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.361705 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" event={"ID":"e75c1729-dbe8-4486-a74d-fa6611610662","Type":"ContainerDied","Data":"026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c"} Oct 30 02:55:31.361827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.361761 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-qfc9q" event={"ID":"e75c1729-dbe8-4486-a74d-fa6611610662","Type":"ContainerDied","Data":"f70114b3e62b18859139db173759519eba73123ebd3a8e1680022b96322f2029"} Oct 30 02:55:31.361827 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.361784 6275 scope.go:122] "RemoveContainer" containerID="026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c" Oct 30 02:55:31.370826 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.370809 6275 scope.go:122] "RemoveContainer" containerID="026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c" Oct 30 02:55:31.371085 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:55:31.371064 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c\": container with ID starting with 026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c not found: ID does not exist" containerID="026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c" Oct 30 02:55:31.371142 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.371093 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c"} err="failed to get container status \"026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c\": rpc error: code = NotFound desc = could not find container \"026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c\": container with ID starting with 026660a42c84a4dbef37ca012e38df5cff2fe7485d2cc957b65d01777c2e241c not found: ID does not exist" Oct 30 02:55:31.407319 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.407294 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-qfc9q"] Oct 30 02:55:31.417810 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.417788 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-qfc9q"] Oct 30 02:55:31.738878 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:31.738847 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75c1729-dbe8-4486-a74d-fa6611610662" path="/var/lib/kubelet/pods/e75c1729-dbe8-4486-a74d-fa6611610662/volumes" Oct 30 02:55:39.353644 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:39.353611 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-75c985bd9-6thmd" Oct 30 02:55:39.850479 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:39.850448 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24"] Oct 30 02:55:43.025596 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.025557 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-pnxwc"] Oct 30 02:55:43.026184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.026001 6275 cpu_manager.go:414] "containerMap: removing container" podUID="e75c1729-dbe8-4486-a74d-fa6611610662" containerName="payload-processing" Oct 30 02:55:43.026184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.026019 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75c1729-dbe8-4486-a74d-fa6611610662" containerName="payload-processing" Oct 30 02:55:43.026184 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.026176 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75c1729-dbe8-4486-a74d-fa6611610662" containerName="payload-processing" Oct 30 02:55:43.030781 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.030758 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.045350 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.045323 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Oct 30 02:55:43.056563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.056475 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:55:43.056563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.056491 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hjgkp\"" Oct 30 02:55:43.056563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.056478 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:55:43.056563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.056521 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:55:43.056563 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.056553 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:55:43.058483 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.058457 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-pnxwc"] Oct 30 02:55:43.104675 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.104630 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.105094 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.105055 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.105309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.105111 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.105309 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.105171 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbqr\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.205733 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.205699 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.205926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.205738 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.205926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.205773 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbqr\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.205926 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.205838 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.206527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.206506 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.206685 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.206665 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.208382 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.208358 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.227456 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.227430 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbqr\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr\") pod \"payload-processing-external-model-758758b45d-pnxwc\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.344352 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.344264 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:43.647527 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:43.647440 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-pnxwc"] Oct 30 02:55:43.652022 ip-10-0-143-27 kubenswrapper[6275]: W1030 02:55:43.651995 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06854894_893b_4edf_9c53_75aeb48f4222.slice/crio-45e3952fe385a4e937e24034b08b8c25bbdaee13015367900945eb8b28b41a61 WatchSource:0}: Error finding container 45e3952fe385a4e937e24034b08b8c25bbdaee13015367900945eb8b28b41a61: Status 404 returned error can't find the container with id 45e3952fe385a4e937e24034b08b8c25bbdaee13015367900945eb8b28b41a61 Oct 30 02:55:44.448144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:44.447437 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" event={"ID":"06854894-893b-4edf-9c53-75aeb48f4222","Type":"ContainerStarted","Data":"03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278"} Oct 30 02:55:44.448144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:44.447539 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" event={"ID":"06854894-893b-4edf-9c53-75aeb48f4222","Type":"ContainerStarted","Data":"45e3952fe385a4e937e24034b08b8c25bbdaee13015367900945eb8b28b41a61"} Oct 30 02:55:44.448144 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:44.447592 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:55:44.491152 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:44.489345 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" podStartSLOduration=1.489326938 podStartE2EDuration="1.489326938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:44.488531906 +0000 UTC m=+5172.880339540" watchObservedRunningTime="2026-10-30 02:55:44.489326938 +0000 UTC m=+5172.881134571" Oct 30 02:55:44.851613 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:44.851557 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" podUID="d7f14e0d-dd87-456a-88b2-c2c9a762403c" containerName="payload-pre-processing" containerID="cri-o://f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9" gracePeriod=25 Oct 30 02:55:45.139273 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.139238 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:55:45.231730 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.231689 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6\" (UniqueName: \"kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6\") pod \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " Oct 30 02:55:45.231956 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.231777 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume\") pod \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\" (UID: \"d7f14e0d-dd87-456a-88b2-c2c9a762403c\") " Oct 30 02:55:45.232223 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.232195 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume" pod "d7f14e0d-dd87-456a-88b2-c2c9a762403c" (UID: "d7f14e0d-dd87-456a-88b2-c2c9a762403c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:55:45.234088 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.234045 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6" pod "d7f14e0d-dd87-456a-88b2-c2c9a762403c" (UID: "d7f14e0d-dd87-456a-88b2-c2c9a762403c"). InnerVolumeSpecName "kube-api-access-98xk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:55:45.333214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.333160 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xk6\" (UniqueName: \"kubernetes.io/projected/d7f14e0d-dd87-456a-88b2-c2c9a762403c-kube-api-access-98xk6\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:55:45.333214 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.333207 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f14e0d-dd87-456a-88b2-c2c9a762403c-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:55:45.454585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.454475 6275 generic.go:356] "Generic (PLEG): container finished" podID="d7f14e0d-dd87-456a-88b2-c2c9a762403c" containerID="f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9" exitCode=0 Oct 30 02:55:45.454585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.454513 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" Oct 30 02:55:45.454585 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.454560 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" event={"ID":"d7f14e0d-dd87-456a-88b2-c2c9a762403c","Type":"ContainerDied","Data":"f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9"} Oct 30 02:55:45.455271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.454623 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24" event={"ID":"d7f14e0d-dd87-456a-88b2-c2c9a762403c","Type":"ContainerDied","Data":"481b3dddf6484c1f98a47ab294bfff83029281a8989f476dd8d2d0760197e1b0"} Oct 30 02:55:45.455271 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.454662 6275 scope.go:122] "RemoveContainer" containerID="f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9" Oct 30 02:55:45.467198 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.467168 6275 scope.go:122] "RemoveContainer" containerID="f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9" Oct 30 02:55:45.467521 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:55:45.467491 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9\": container with ID starting with f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9 not found: ID does not exist" containerID="f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9" Oct 30 02:55:45.467662 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.467532 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9"} err="failed to get container status \"f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9\": rpc error: code = NotFound desc = could not find container \"f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9\": container with ID starting with f0ad3158da6f7298168b7227732314c834ce10eac30abc0f0230d89cf04ddae9 not found: ID does not exist" Oct 30 02:55:45.502640 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.502297 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24"] Oct 30 02:55:45.513920 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.513886 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-8dd7b7cc5-ngv24"] Oct 30 02:55:45.737179 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:45.737095 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f14e0d-dd87-456a-88b2-c2c9a762403c" path="/var/lib/kubelet/pods/d7f14e0d-dd87-456a-88b2-c2c9a762403c/volumes" Oct 30 02:55:46.729958 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:55:46.729918 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:55:55.457574 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:55:55.457543 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:56:16.696183 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:56:16.696142 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:56:46.696780 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:56:46.696703 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:57:16.697055 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:57:16.697014 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:57:34.028657 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:34.028621 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-pnxwc"] Oct 30 02:57:39.029410 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.029357 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" podUID="06854894-893b-4edf-9c53-75aeb48f4222" containerName="payload-processing" containerID="cri-o://03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278" gracePeriod=25 Oct 30 02:57:39.284100 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.284042 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:57:39.349028 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.348997 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials\") pod \"06854894-893b-4edf-9c53-75aeb48f4222\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " Oct 30 02:57:39.349232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.349072 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume\") pod \"06854894-893b-4edf-9c53-75aeb48f4222\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " Oct 30 02:57:39.349232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.349112 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume\") pod \"06854894-893b-4edf-9c53-75aeb48f4222\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " Oct 30 02:57:39.349232 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.349158 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr\") pod \"06854894-893b-4edf-9c53-75aeb48f4222\" (UID: \"06854894-893b-4edf-9c53-75aeb48f4222\") " Oct 30 02:57:39.349557 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.349525 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume" pod "06854894-893b-4edf-9c53-75aeb48f4222" (UID: "06854894-893b-4edf-9c53-75aeb48f4222"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:39.349671 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.349535 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume" pod "06854894-893b-4edf-9c53-75aeb48f4222" (UID: "06854894-893b-4edf-9c53-75aeb48f4222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:39.351485 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.351458 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr" pod "06854894-893b-4edf-9c53-75aeb48f4222" (UID: "06854894-893b-4edf-9c53-75aeb48f4222"). InnerVolumeSpecName "kube-api-access-hpbqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:39.351777 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.351746 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials" pod "06854894-893b-4edf-9c53-75aeb48f4222" (UID: "06854894-893b-4edf-9c53-75aeb48f4222"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:39.450780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.450747 6275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-provider-credentials\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:57:39.450780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.450775 6275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-routing-overlay-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:57:39.450780 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.450784 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06854894-893b-4edf-9c53-75aeb48f4222-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:57:39.450995 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:39.450792 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpbqr\" (UniqueName: \"kubernetes.io/projected/06854894-893b-4edf-9c53-75aeb48f4222-kube-api-access-hpbqr\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 02:57:40.020253 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.020218 6275 generic.go:356] "Generic (PLEG): container finished" podID="06854894-893b-4edf-9c53-75aeb48f4222" containerID="03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278" exitCode=0 Oct 30 02:57:40.020444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.020268 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" Oct 30 02:57:40.020444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.020307 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" event={"ID":"06854894-893b-4edf-9c53-75aeb48f4222","Type":"ContainerDied","Data":"03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278"} Oct 30 02:57:40.020444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.020363 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-pnxwc" event={"ID":"06854894-893b-4edf-9c53-75aeb48f4222","Type":"ContainerDied","Data":"45e3952fe385a4e937e24034b08b8c25bbdaee13015367900945eb8b28b41a61"} Oct 30 02:57:40.020444 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.020387 6275 scope.go:122] "RemoveContainer" containerID="03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278" Oct 30 02:57:40.029131 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.029099 6275 scope.go:122] "RemoveContainer" containerID="03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278" Oct 30 02:57:40.029384 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:57:40.029364 6275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278\": container with ID starting with 03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278 not found: ID does not exist" containerID="03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278" Oct 30 02:57:40.029466 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.029397 6275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278"} err="failed to get container status \"03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278\": rpc error: code = NotFound desc = could not find container \"03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278\": container with ID starting with 03d93e764ac7a4fee750da3254df98b7de28ca3494b46f7a1becedf2c8e70278 not found: ID does not exist" Oct 30 02:57:40.066951 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.066906 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-pnxwc"] Oct 30 02:57:40.076439 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:40.076414 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-pnxwc"] Oct 30 02:57:41.737045 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:57:41.737010 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06854894-893b-4edf-9c53-75aeb48f4222" path="/var/lib/kubelet/pods/06854894-893b-4edf-9c53-75aeb48f4222/volumes" Oct 30 02:57:46.695650 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:57:46.695606 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:58:05.101147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:58:05.100255 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:58:05.101147 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:58:05.100359 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:58:16.696644 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:58:16.696561 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:58:46.696849 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:58:46.696807 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:59:16.696161 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:59:16.696097 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 02:59:32.874073 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:59:32.874039 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:59:32.893143 ip-10-0-143-27 kubenswrapper[6275]: I1030 02:59:32.893095 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 02:59:46.695980 ip-10-0-143-27 kubenswrapper[6275]: E1030 02:59:46.695884 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:00:16.696352 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:00:16.696313 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:00:31.137342 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:00:31.137296 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:00:31.137766 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:00:31.137368 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:00:35.096274 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:00:35.096224 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:00:35.096837 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:00:35.096314 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:00:46.695855 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:00:46.695812 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:01:16.696419 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:01:16.696334 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:01:46.696893 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:01:46.696853 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:01:52.182988 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.182943 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4"] Oct 30 03:01:52.183572 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.183558 6275 cpu_manager.go:414] "containerMap: removing container" podUID="d7f14e0d-dd87-456a-88b2-c2c9a762403c" containerName="payload-pre-processing" Oct 30 03:01:52.183622 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.183582 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f14e0d-dd87-456a-88b2-c2c9a762403c" containerName="payload-pre-processing" Oct 30 03:01:52.183622 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.183592 6275 cpu_manager.go:414] "containerMap: removing container" podUID="06854894-893b-4edf-9c53-75aeb48f4222" containerName="payload-processing" Oct 30 03:01:52.183622 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.183597 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06854894-893b-4edf-9c53-75aeb48f4222" containerName="payload-processing" Oct 30 03:01:52.183768 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.183682 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f14e0d-dd87-456a-88b2-c2c9a762403c" containerName="payload-pre-processing" Oct 30 03:01:52.183768 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.183689 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="06854894-893b-4edf-9c53-75aeb48f4222" containerName="payload-processing" Oct 30 03:01:52.186644 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.186622 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.205269 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.205238 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-447d90-serving-cert\"" Oct 30 03:01:52.216680 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.216652 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4"] Oct 30 03:01:52.284887 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.284851 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.285070 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.284972 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.285070 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.285017 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxnf\" (UniqueName: \"kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.285194 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.285134 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.385812 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.385771 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.385812 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.385817 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxnf\" (UniqueName: \"kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.386046 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.385881 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.386046 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.385918 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.386046 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:01:52.385956 6275 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-serving-cert: secret "maas-api-e2e-worker-single-main-447d90-serving-cert" not found Oct 30 03:01:52.386046 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:01:52.386034 6275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls podName:5764dcbe-aeef-4605-b7e7-8d610cb395a8 nodeName:}" failed. No retries permitted until 2026-10-30 03:01:52.886013278 +0000 UTC m=+5541.277820889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls") pod "maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" (UID: "5764dcbe-aeef-4605-b7e7-8d610cb395a8") : secret "maas-api-e2e-worker-single-main-447d90-serving-cert" not found Oct 30 03:01:52.386712 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.386690 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.388471 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.388452 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.407999 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.407969 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxnf\" (UniqueName: \"kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.722076 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.722036 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh"] Oct 30 03:01:52.725844 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.725822 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:52.749044 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.749001 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh"] Oct 30 03:01:52.784920 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.784881 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8944a547-dockercfg-n2bld\"" Oct 30 03:01:52.785248 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.785011 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-447d90\"" Oct 30 03:01:52.790385 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.790329 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:52.790593 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.790440 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzngx\" (UniqueName: \"kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx\") pod \"payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:52.891204 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.891166 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.891204 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.891207 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:52.891451 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.891266 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzngx\" (UniqueName: \"kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx\") pod \"payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:52.891998 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.891972 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:52.893817 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.893792 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:52.913108 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:52.913077 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzngx\" (UniqueName: \"kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx\") pod \"payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:53.039989 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.039950 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:53.102791 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.102755 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:53.347500 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.347463 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh"] Oct 30 03:01:53.349274 ip-10-0-143-27 kubenswrapper[6275]: W1030 03:01:53.349247 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cae73b3_d319_4457_9e13_71c317197e69.slice/crio-5b11a5489121840a19a51e3de4ab4f11d887f137a57291efeaa90ec77b9103bb WatchSource:0}: Error finding container 5b11a5489121840a19a51e3de4ab4f11d887f137a57291efeaa90ec77b9103bb: Status 404 returned error can't find the container with id 5b11a5489121840a19a51e3de4ab4f11d887f137a57291efeaa90ec77b9103bb Oct 30 03:01:53.421109 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.421079 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4"] Oct 30 03:01:53.424865 ip-10-0-143-27 kubenswrapper[6275]: W1030 03:01:53.424842 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5764dcbe_aeef_4605_b7e7_8d610cb395a8.slice/crio-444328fc52e23e9a224655e1d000c833139c98608e605bb0524e3dae3e5e98fa WatchSource:0}: Error finding container 444328fc52e23e9a224655e1d000c833139c98608e605bb0524e3dae3e5e98fa: Status 404 returned error can't find the container with id 444328fc52e23e9a224655e1d000c833139c98608e605bb0524e3dae3e5e98fa Oct 30 03:01:53.426322 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.426302 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:01:53.454378 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.454347 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" event={"ID":"9cae73b3-d319-4457-9e13-71c317197e69","Type":"ContainerStarted","Data":"c8842721b0a5b6b920cfb70135fff555918101858e665d94d8db9fb804bb628e"} Oct 30 03:01:53.454512 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.454405 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" event={"ID":"9cae73b3-d319-4457-9e13-71c317197e69","Type":"ContainerStarted","Data":"5b11a5489121840a19a51e3de4ab4f11d887f137a57291efeaa90ec77b9103bb"} Oct 30 03:01:53.454582 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.454536 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:01:53.455669 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.455644 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" event={"ID":"5764dcbe-aeef-4605-b7e7-8d610cb395a8","Type":"ContainerStarted","Data":"444328fc52e23e9a224655e1d000c833139c98608e605bb0524e3dae3e5e98fa"} Oct 30 03:01:53.492151 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:53.492090 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" podStartSLOduration=1.4920774620000001 podStartE2EDuration="1.492077462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:53.490968197 +0000 UTC m=+5541.882776035" watchObservedRunningTime="2026-10-30 03:01:53.492077462 +0000 UTC m=+5541.883885097" Oct 30 03:01:54.462371 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:54.462336 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" event={"ID":"5764dcbe-aeef-4605-b7e7-8d610cb395a8","Type":"ContainerStarted","Data":"658a3cdca47a3db4b28d1514d3037c84ff5bcba1e4d763073a960d5d783a408f"} Oct 30 03:01:54.462842 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:54.462545 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:01:54.502311 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:01:54.502268 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" podStartSLOduration=2.086610508 podStartE2EDuration="2.502254176s" totalImagesPullingTime="415.643668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:54.501117151 +0000 UTC m=+5542.892924784" watchObservedRunningTime="2026-10-30 03:01:54.502254176 +0000 UTC m=+5542.894061808" Oct 30 03:02:00.480926 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:00.480898 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:02:04.465773 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:04.465741 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:02:16.697356 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:02:16.697317 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:02:46.695774 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:02:46.695694 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:02:57.426472 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:57.426428 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh"] Oct 30 03:02:58.388871 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.388827 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4"] Oct 30 03:02:58.389167 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.389093 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" podUID="5764dcbe-aeef-4605-b7e7-8d610cb395a8" containerName="maas-api" containerID="cri-o://658a3cdca47a3db4b28d1514d3037c84ff5bcba1e4d763073a960d5d783a408f" gracePeriod=30 Oct 30 03:02:58.765315 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.765171 6275 generic.go:356] "Generic (PLEG): container finished" podID="5764dcbe-aeef-4605-b7e7-8d610cb395a8" containerID="658a3cdca47a3db4b28d1514d3037c84ff5bcba1e4d763073a960d5d783a408f" exitCode=0 Oct 30 03:02:58.765315 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.765268 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" event={"ID":"5764dcbe-aeef-4605-b7e7-8d610cb395a8","Type":"ContainerDied","Data":"658a3cdca47a3db4b28d1514d3037c84ff5bcba1e4d763073a960d5d783a408f"} Oct 30 03:02:58.867144 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.866054 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:02:58.911379 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.911352 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf\" (UniqueName: \"kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf\") pod \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " Oct 30 03:02:58.911578 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.911414 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs\") pod \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " Oct 30 03:02:58.911578 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.911498 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca\") pod \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " Oct 30 03:02:58.911578 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.911559 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\") pod \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\" (UID: \"5764dcbe-aeef-4605-b7e7-8d610cb395a8\") " Oct 30 03:02:58.911950 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.911904 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca" pod "5764dcbe-aeef-4605-b7e7-8d610cb395a8" (UID: "5764dcbe-aeef-4605-b7e7-8d610cb395a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:02:58.913823 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.913786 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf" pod "5764dcbe-aeef-4605-b7e7-8d610cb395a8" (UID: "5764dcbe-aeef-4605-b7e7-8d610cb395a8"). InnerVolumeSpecName "kube-api-access-npxnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:02:58.913823 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.913804 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls" pod "5764dcbe-aeef-4605-b7e7-8d610cb395a8" (UID: "5764dcbe-aeef-4605-b7e7-8d610cb395a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:02:58.913980 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:58.913879 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs" pod "5764dcbe-aeef-4605-b7e7-8d610cb395a8" (UID: "5764dcbe-aeef-4605-b7e7-8d610cb395a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:02:59.012606 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.012576 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:02:59.012606 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.012603 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5764dcbe-aeef-4605-b7e7-8d610cb395a8-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:02:59.012809 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.012614 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5764dcbe-aeef-4605-b7e7-8d610cb395a8-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:02:59.012809 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.012623 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxnf\" (UniqueName: \"kubernetes.io/projected/5764dcbe-aeef-4605-b7e7-8d610cb395a8-kube-api-access-npxnf\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:02:59.770887 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.770846 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" event={"ID":"5764dcbe-aeef-4605-b7e7-8d610cb395a8","Type":"ContainerDied","Data":"444328fc52e23e9a224655e1d000c833139c98608e605bb0524e3dae3e5e98fa"} Oct 30 03:02:59.771290 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.770910 6275 scope.go:122] "RemoveContainer" containerID="658a3cdca47a3db4b28d1514d3037c84ff5bcba1e4d763073a960d5d783a408f" Oct 30 03:02:59.771290 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.770867 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4" Oct 30 03:02:59.818484 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.818454 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4"] Oct 30 03:02:59.828877 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:02:59.828849 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-447d90-57b8644c85-wgbf4"] Oct 30 03:03:01.736699 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:01.736664 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5764dcbe-aeef-4605-b7e7-8d610cb395a8" path="/var/lib/kubelet/pods/5764dcbe-aeef-4605-b7e7-8d610cb395a8/volumes" Oct 30 03:03:02.427733 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:02.427678 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" podUID="9cae73b3-d319-4457-9e13-71c317197e69" containerName="payload-pre-processing" containerID="cri-o://c8842721b0a5b6b920cfb70135fff555918101858e665d94d8db9fb804bb628e" gracePeriod=25 Oct 30 03:03:04.464011 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:04.463966 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" podUID="9cae73b3-d319-4457-9e13-71c317197e69" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:03:14.464273 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:14.464232 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" podUID="9cae73b3-d319-4457-9e13-71c317197e69" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:03:16.695118 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:03:16.695042 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:03:22.507146 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:03:22.506742 6275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cae73b3_d319_4457_9e13_71c317197e69.slice/crio-conmon-c8842721b0a5b6b920cfb70135fff555918101858e665d94d8db9fb804bb628e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 03:03:22.924791 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:22.924763 6275 generic.go:356] "Generic (PLEG): container finished" podID="9cae73b3-d319-4457-9e13-71c317197e69" containerID="c8842721b0a5b6b920cfb70135fff555918101858e665d94d8db9fb804bb628e" exitCode=0 Oct 30 03:03:22.924942 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:22.924835 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" event={"ID":"9cae73b3-d319-4457-9e13-71c317197e69","Type":"ContainerDied","Data":"c8842721b0a5b6b920cfb70135fff555918101858e665d94d8db9fb804bb628e"} Oct 30 03:03:22.924942 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:22.924890 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" event={"ID":"9cae73b3-d319-4457-9e13-71c317197e69","Type":"ContainerDied","Data":"5b11a5489121840a19a51e3de4ab4f11d887f137a57291efeaa90ec77b9103bb"} Oct 30 03:03:22.924942 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:22.924907 6275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b11a5489121840a19a51e3de4ab4f11d887f137a57291efeaa90ec77b9103bb" Oct 30 03:03:22.934706 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:22.934681 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:03:23.022990 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.022760 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx\" (UniqueName: \"kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx\") pod \"9cae73b3-d319-4457-9e13-71c317197e69\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " Oct 30 03:03:23.022990 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.022869 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume\") pod \"9cae73b3-d319-4457-9e13-71c317197e69\" (UID: \"9cae73b3-d319-4457-9e13-71c317197e69\") " Oct 30 03:03:23.023513 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.023482 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume" pod "9cae73b3-d319-4457-9e13-71c317197e69" (UID: "9cae73b3-d319-4457-9e13-71c317197e69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:23.024974 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.024947 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx" pod "9cae73b3-d319-4457-9e13-71c317197e69" (UID: "9cae73b3-d319-4457-9e13-71c317197e69"). InnerVolumeSpecName "kube-api-access-dzngx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:23.124383 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.124323 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzngx\" (UniqueName: \"kubernetes.io/projected/9cae73b3-d319-4457-9e13-71c317197e69-kube-api-access-dzngx\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:03:23.124383 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.124352 6275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae73b3-d319-4457-9e13-71c317197e69-plugins-config-volume\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:03:23.929328 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.929292 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh" Oct 30 03:03:23.975028 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.974993 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh"] Oct 30 03:03:23.985372 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:23.985334 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-447d90-949866l2jh"] Oct 30 03:03:25.737973 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:03:25.737939 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cae73b3-d319-4457-9e13-71c317197e69" path="/var/lib/kubelet/pods/9cae73b3-d319-4457-9e13-71c317197e69/volumes" Oct 30 03:03:46.696067 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:03:46.696030 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:04:16.695686 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:04:16.695608 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:04:32.938254 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:04:32.938227 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 03:04:32.943109 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:04:32.943087 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 03:04:46.696480 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:04:46.696436 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:05:16.696057 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:05:16.696017 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:05:21.349327 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349294 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf"] Oct 30 03:05:21.349802 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349776 6275 cpu_manager.go:414] "containerMap: removing container" podUID="5764dcbe-aeef-4605-b7e7-8d610cb395a8" containerName="maas-api" Oct 30 03:05:21.349802 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349795 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5764dcbe-aeef-4605-b7e7-8d610cb395a8" containerName="maas-api" Oct 30 03:05:21.349944 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349831 6275 cpu_manager.go:414] "containerMap: removing container" podUID="9cae73b3-d319-4457-9e13-71c317197e69" containerName="payload-pre-processing" Oct 30 03:05:21.349944 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349842 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cae73b3-d319-4457-9e13-71c317197e69" containerName="payload-pre-processing" Oct 30 03:05:21.350028 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349949 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cae73b3-d319-4457-9e13-71c317197e69" containerName="payload-pre-processing" Oct 30 03:05:21.350028 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.349972 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5764dcbe-aeef-4605-b7e7-8d610cb395a8" containerName="maas-api" Oct 30 03:05:21.352263 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.352242 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.382502 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.382466 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf"] Oct 30 03:05:21.420606 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.420550 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d75b5f-856e-4d57-927f-126dc33367f1-metrics-certs\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.420822 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.420619 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d75b5f-856e-4d57-927f-126dc33367f1-maas-api-tls\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.420822 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.420773 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d75b5f-856e-4d57-927f-126dc33367f1-openshift-service-ca\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.420967 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.420946 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2kh\" (UniqueName: \"kubernetes.io/projected/52d75b5f-856e-4d57-927f-126dc33367f1-kube-api-access-rv2kh\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.522248 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.522215 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2kh\" (UniqueName: \"kubernetes.io/projected/52d75b5f-856e-4d57-927f-126dc33367f1-kube-api-access-rv2kh\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.522457 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.522264 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d75b5f-856e-4d57-927f-126dc33367f1-metrics-certs\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.522457 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.522296 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d75b5f-856e-4d57-927f-126dc33367f1-maas-api-tls\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.522457 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.522341 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d75b5f-856e-4d57-927f-126dc33367f1-openshift-service-ca\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.522989 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.522968 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d75b5f-856e-4d57-927f-126dc33367f1-openshift-service-ca\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.524814 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.524795 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d75b5f-856e-4d57-927f-126dc33367f1-maas-api-tls\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.524921 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.524901 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d75b5f-856e-4d57-927f-126dc33367f1-metrics-certs\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.543498 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.543473 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2kh\" (UniqueName: \"kubernetes.io/projected/52d75b5f-856e-4d57-927f-126dc33367f1-kube-api-access-rv2kh\") pod \"maas-api-6cf77bf7d-dm6cf\" (UID: \"52d75b5f-856e-4d57-927f-126dc33367f1\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.666467 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.666390 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:21.969161 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:21.969072 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf"] Oct 30 03:05:21.973712 ip-10-0-143-27 kubenswrapper[6275]: W1030 03:05:21.973684 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d75b5f_856e_4d57_927f_126dc33367f1.slice/crio-25bbec912ae0df7a181b208a87741c0fbcaa63c9b470fdd867c90d116b73c488 WatchSource:0}: Error finding container 25bbec912ae0df7a181b208a87741c0fbcaa63c9b470fdd867c90d116b73c488: Status 404 returned error can't find the container with id 25bbec912ae0df7a181b208a87741c0fbcaa63c9b470fdd867c90d116b73c488 Oct 30 03:05:22.478663 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:22.478627 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" event={"ID":"52d75b5f-856e-4d57-927f-126dc33367f1","Type":"ContainerStarted","Data":"9a04ac7624e20991a6af3f3bbcfa708012958e3e7db61015d9e0cb7248cde711"} Oct 30 03:05:22.479105 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:22.478691 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" event={"ID":"52d75b5f-856e-4d57-927f-126dc33367f1","Type":"ContainerStarted","Data":"25bbec912ae0df7a181b208a87741c0fbcaa63c9b470fdd867c90d116b73c488"} Oct 30 03:05:22.479105 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:22.478721 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:22.518504 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:22.518442 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" podStartSLOduration=1.099260937 podStartE2EDuration="1.518427306s" totalImagesPullingTime="419.166369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:22.51690301 +0000 UTC m=+5750.908710834" watchObservedRunningTime="2026-10-30 03:05:22.518427306 +0000 UTC m=+5750.910234939" Oct 30 03:05:28.489460 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:28.489429 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-dm6cf" Oct 30 03:05:36.410175 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:36.407403 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2"] Oct 30 03:05:36.410175 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:36.407696 6275 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" podUID="f96f255f-31d7-4bc1-b476-b4e01f0018ca" containerName="maas-api" containerID="cri-o://966ef45a3a6edcee192b6ec57bb8b6c894b0cd67822612e27b7ec8429ba80bfd" gracePeriod=30 Oct 30 03:05:37.620085 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:37.620033 6275 generic.go:356] "Generic (PLEG): container finished" podID="f96f255f-31d7-4bc1-b476-b4e01f0018ca" containerID="966ef45a3a6edcee192b6ec57bb8b6c894b0cd67822612e27b7ec8429ba80bfd" exitCode=0 Oct 30 03:05:37.620630 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:37.620145 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" event={"ID":"f96f255f-31d7-4bc1-b476-b4e01f0018ca","Type":"ContainerDied","Data":"966ef45a3a6edcee192b6ec57bb8b6c894b0cd67822612e27b7ec8429ba80bfd"} Oct 30 03:05:37.931152 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:37.931109 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 03:05:38.054504 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.054459 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925\" (UniqueName: \"kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925\") pod \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " Oct 30 03:05:38.054869 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.054847 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls\") pod \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " Oct 30 03:05:38.055037 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.055019 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca\") pod \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " Oct 30 03:05:38.055221 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.055205 6275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs\") pod \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\" (UID: \"f96f255f-31d7-4bc1-b476-b4e01f0018ca\") " Oct 30 03:05:38.073796 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.073519 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca" pod "f96f255f-31d7-4bc1-b476-b4e01f0018ca" (UID: "f96f255f-31d7-4bc1-b476-b4e01f0018ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:38.086141 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.081029 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs" pod "f96f255f-31d7-4bc1-b476-b4e01f0018ca" (UID: "f96f255f-31d7-4bc1-b476-b4e01f0018ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:38.086141 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.083562 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925" pod "f96f255f-31d7-4bc1-b476-b4e01f0018ca" (UID: "f96f255f-31d7-4bc1-b476-b4e01f0018ca"). InnerVolumeSpecName "kube-api-access-55925". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:05:38.086510 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.086489 6275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls" pod "f96f255f-31d7-4bc1-b476-b4e01f0018ca" (UID: "f96f255f-31d7-4bc1-b476-b4e01f0018ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:38.156551 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.156515 6275 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96f255f-31d7-4bc1-b476-b4e01f0018ca-openshift-service-ca\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:05:38.156825 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.156806 6275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-metrics-certs\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:05:38.156947 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.156933 6275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55925\" (UniqueName: \"kubernetes.io/projected/f96f255f-31d7-4bc1-b476-b4e01f0018ca-kube-api-access-55925\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:05:38.157047 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.157035 6275 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96f255f-31d7-4bc1-b476-b4e01f0018ca-maas-api-tls\") on node \"ip-10-0-143-27.ec2.internal\" DevicePath \"\"" Oct 30 03:05:38.625414 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.625388 6275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" Oct 30 03:05:38.625884 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.625424 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2" event={"ID":"f96f255f-31d7-4bc1-b476-b4e01f0018ca","Type":"ContainerDied","Data":"97bc18e7fc8870304b9cdf055db6c261772335348041c17e99994420f322bc02"} Oct 30 03:05:38.625884 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.625486 6275 scope.go:122] "RemoveContainer" containerID="966ef45a3a6edcee192b6ec57bb8b6c894b0cd67822612e27b7ec8429ba80bfd" Oct 30 03:05:38.671966 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.671935 6275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2"] Oct 30 03:05:38.682380 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:38.682351 6275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-s6tp2"] Oct 30 03:05:39.736586 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:05:39.736554 6275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96f255f-31d7-4bc1-b476-b4e01f0018ca" path="/var/lib/kubelet/pods/f96f255f-31d7-4bc1-b476-b4e01f0018ca/volumes" Oct 30 03:05:46.697191 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:05:46.697087 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:06:16.695894 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:06:16.695857 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:06:31.131298 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:06:31.131257 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lxtlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:06:31.131821 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:06:31.131330 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lxtlt" podUID="b4fb616c-0b22-482d-86e1-dade8fda8955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:06:35.103145 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:06:35.101215 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:06:35.103145 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:06:35.101324 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:06:46.695833 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:06:46.695792 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:07:16.696375 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:07:16.696294 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:07:46.695739 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:07:46.695699 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:08:05.100146 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:08:05.099203 6275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-nzg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:08:05.100146 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:08:05.099272 6275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-nzg5g" podUID="df283d5f-6e54-406a-b11a-9da1af27074b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:08:16.701210 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:08:16.701170 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:08:36.501835 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:08:36.501803 6275 scope.go:122] "RemoveContainer" containerID="c8842721b0a5b6b920cfb70135fff555918101858e665d94d8db9fb804bb628e" Oct 30 03:08:46.732904 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:08:46.732825 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:09:16.699346 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:09:16.699297 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:09:32.966551 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:09:32.966520 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 03:09:32.972705 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:09:32.972681 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kd25_0f0245e1-cf5c-4d84-8072-bdd72bfe3f36/ovn-acl-logging/0.log" Oct 30 03:09:46.696978 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:09:46.696942 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:10:16.695546 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:10:16.695467 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:10:28.793327 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:28.793292 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7795bf7868-gmhpg_09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3/authorino/0.log" Oct 30 03:10:41.183517 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:41.183480 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:10:42.185501 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:42.185469 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:10:42.841455 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:42.841409 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:10:44.209700 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:44.209663 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cf77bf7d-dm6cf_52d75b5f-856e-4d57-927f-126dc33367f1/maas-api/0.log" Oct 30 03:10:46.695239 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:10:46.695203 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:10:48.001285 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:48.001247 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-b7p6x_7cbd9e1d-6767-4ca5-89d0-aa99a2c89ebf/postgres/0.log" Oct 30 03:10:49.917652 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:49.917615 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7795bf7868-gmhpg_09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3/authorino/0.log" Oct 30 03:10:52.581320 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:52.581285 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5vjwt_e7ec1529-3bf9-4ded-8a38-77e9b1b7964d/discovery/0.log" Oct 30 03:10:52.793117 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:52.793083 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-m9zg8_807e1be9-1186-4cdc-a03b-7739c1720321/kube-auth-proxy/0.log" Oct 30 03:10:53.592954 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:53.592924 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-72kzd_173adfa0-45d6-4a44-b36d-f47bd184592a/istio-proxy/0.log" Oct 30 03:10:54.231642 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:54.231610 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lxtlt_b4fb616c-0b22-482d-86e1-dade8fda8955/istio-proxy/0.log" Oct 30 03:10:54.454274 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:54.454236 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-nzg5g_df283d5f-6e54-406a-b11a-9da1af27074b/istio-proxy/0.log" Oct 30 03:10:55.709869 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:55.709835 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75c985bd9-6thmd_e2362a6d-db63-4753-b0be-625d9d0be0c3/payload-processing/0.log" Oct 30 03:10:56.962603 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:56.962567 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-srspl_3471b390-0a41-44c3-813c-61af6545a43c/main/0.log" Oct 30 03:10:57.147842 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:57.147812 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8mxxx_7bce4f69-7fd4-429a-9420-c6bcfbb52e04/main/0.log" Oct 30 03:10:57.734078 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:57.734049 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82qnb6_56e7d3d4-3f98-40c5-9a45-17237ebb244a/main/0.log" Oct 30 03:10:58.128061 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:10:58.128027 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cpmfd_cac30d76-8aee-4c30-bfe6-66c5a9766419/main/0.log" Oct 30 03:11:09.635316 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.635281 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wd7bs/must-gather-r6s77"] Oct 30 03:11:09.635730 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.635688 6275 cpu_manager.go:414] "containerMap: removing container" podUID="f96f255f-31d7-4bc1-b476-b4e01f0018ca" containerName="maas-api" Oct 30 03:11:09.635730 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.635699 6275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96f255f-31d7-4bc1-b476-b4e01f0018ca" containerName="maas-api" Oct 30 03:11:09.635815 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.635784 6275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96f255f-31d7-4bc1-b476-b4e01f0018ca" containerName="maas-api" Oct 30 03:11:09.638972 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.638954 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.675075 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.675044 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/must-gather-r6s77"] Oct 30 03:11:09.692606 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.692580 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lppxm/must-gather-flqwj"] Oct 30 03:11:09.695414 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.695391 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd7bs\"/\"openshift-service-ca.crt\"" Oct 30 03:11:09.695565 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.695431 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd7bs\"/\"kube-root-ca.crt\"" Oct 30 03:11:09.696366 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.696333 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.700835 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.700814 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlnv\" (UniqueName: \"kubernetes.io/projected/6025f048-fa26-4e36-9550-53bffdea3eb7-kube-api-access-2zlnv\") pod \"must-gather-r6s77\" (UID: \"6025f048-fa26-4e36-9550-53bffdea3eb7\") " pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.700939 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.700848 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6025f048-fa26-4e36-9550-53bffdea3eb7-must-gather-output\") pod \"must-gather-r6s77\" (UID: \"6025f048-fa26-4e36-9550-53bffdea3eb7\") " pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.712158 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.712137 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lppxm\"/\"default-dockercfg-m74jw\"" Oct 30 03:11:09.719745 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.719722 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lppxm/must-gather-flqwj"] Oct 30 03:11:09.747298 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.747251 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lppxm\"/\"kube-root-ca.crt\"" Oct 30 03:11:09.747450 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.747410 6275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lppxm\"/\"openshift-service-ca.crt\"" Oct 30 03:11:09.802331 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.802294 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlnv\" (UniqueName: \"kubernetes.io/projected/6025f048-fa26-4e36-9550-53bffdea3eb7-kube-api-access-2zlnv\") pod \"must-gather-r6s77\" (UID: \"6025f048-fa26-4e36-9550-53bffdea3eb7\") " pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.802516 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.802338 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6025f048-fa26-4e36-9550-53bffdea3eb7-must-gather-output\") pod \"must-gather-r6s77\" (UID: \"6025f048-fa26-4e36-9550-53bffdea3eb7\") " pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.802516 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.802395 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb0560fc-3b1a-4834-aa3c-e69b01f245e0-must-gather-output\") pod \"must-gather-flqwj\" (UID: \"fb0560fc-3b1a-4834-aa3c-e69b01f245e0\") " pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.802516 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.802422 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xlf\" (UniqueName: \"kubernetes.io/projected/fb0560fc-3b1a-4834-aa3c-e69b01f245e0-kube-api-access-48xlf\") pod \"must-gather-flqwj\" (UID: \"fb0560fc-3b1a-4834-aa3c-e69b01f245e0\") " pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.802784 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.802762 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6025f048-fa26-4e36-9550-53bffdea3eb7-must-gather-output\") pod \"must-gather-r6s77\" (UID: \"6025f048-fa26-4e36-9550-53bffdea3eb7\") " pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.825716 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.825682 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlnv\" (UniqueName: \"kubernetes.io/projected/6025f048-fa26-4e36-9550-53bffdea3eb7-kube-api-access-2zlnv\") pod \"must-gather-r6s77\" (UID: \"6025f048-fa26-4e36-9550-53bffdea3eb7\") " pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:09.903060 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.902980 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xlf\" (UniqueName: \"kubernetes.io/projected/fb0560fc-3b1a-4834-aa3c-e69b01f245e0-kube-api-access-48xlf\") pod \"must-gather-flqwj\" (UID: \"fb0560fc-3b1a-4834-aa3c-e69b01f245e0\") " pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.903248 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.903115 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb0560fc-3b1a-4834-aa3c-e69b01f245e0-must-gather-output\") pod \"must-gather-flqwj\" (UID: \"fb0560fc-3b1a-4834-aa3c-e69b01f245e0\") " pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.903447 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.903425 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb0560fc-3b1a-4834-aa3c-e69b01f245e0-must-gather-output\") pod \"must-gather-flqwj\" (UID: \"fb0560fc-3b1a-4834-aa3c-e69b01f245e0\") " pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.926259 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.926233 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xlf\" (UniqueName: \"kubernetes.io/projected/fb0560fc-3b1a-4834-aa3c-e69b01f245e0-kube-api-access-48xlf\") pod \"must-gather-flqwj\" (UID: \"fb0560fc-3b1a-4834-aa3c-e69b01f245e0\") " pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:09.953184 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:09.953157 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/must-gather-r6s77" Oct 30 03:11:10.010420 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:10.010391 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lppxm/must-gather-flqwj" Oct 30 03:11:10.253567 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:10.253537 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/must-gather-r6s77"] Oct 30 03:11:10.255938 ip-10-0-143-27 kubenswrapper[6275]: W1030 03:11:10.255905 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6025f048_fa26_4e36_9550_53bffdea3eb7.slice/crio-ca3e8687b3a890f828b43fdbc102d41e684ee190a048211f9d0653a585d5b8d1 WatchSource:0}: Error finding container ca3e8687b3a890f828b43fdbc102d41e684ee190a048211f9d0653a585d5b8d1: Status 404 returned error can't find the container with id ca3e8687b3a890f828b43fdbc102d41e684ee190a048211f9d0653a585d5b8d1 Oct 30 03:11:10.257642 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:10.257625 6275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:11:10.317703 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:10.317629 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lppxm/must-gather-flqwj"] Oct 30 03:11:10.319787 ip-10-0-143-27 kubenswrapper[6275]: W1030 03:11:10.319762 6275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb0560fc_3b1a_4834_aa3c_e69b01f245e0.slice/crio-78a3f4d70bd246992fe9b024f5ef0a934b2e77d26fefb525e9f767c83aa89825 WatchSource:0}: Error finding container 78a3f4d70bd246992fe9b024f5ef0a934b2e77d26fefb525e9f767c83aa89825: Status 404 returned error can't find the container with id 78a3f4d70bd246992fe9b024f5ef0a934b2e77d26fefb525e9f767c83aa89825 Oct 30 03:11:10.500771 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:10.500731 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/must-gather-r6s77" event={"ID":"6025f048-fa26-4e36-9550-53bffdea3eb7","Type":"ContainerStarted","Data":"ca3e8687b3a890f828b43fdbc102d41e684ee190a048211f9d0653a585d5b8d1"} Oct 30 03:11:10.501850 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:10.501820 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lppxm/must-gather-flqwj" event={"ID":"fb0560fc-3b1a-4834-aa3c-e69b01f245e0","Type":"ContainerStarted","Data":"78a3f4d70bd246992fe9b024f5ef0a934b2e77d26fefb525e9f767c83aa89825"} Oct 30 03:11:11.511814 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:11.511754 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/must-gather-r6s77" event={"ID":"6025f048-fa26-4e36-9550-53bffdea3eb7","Type":"ContainerStarted","Data":"a25c40d9e3b7474a566eb1cbc60b75c432f88e893d8fa8eefc8ccefbfda9e669"} Oct 30 03:11:12.521186 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:12.520787 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/must-gather-r6s77" event={"ID":"6025f048-fa26-4e36-9550-53bffdea3eb7","Type":"ContainerStarted","Data":"2c91d9d1454f29d67dfd234ec62f2d29a6571b5585068d4a8e325f6eeb32dcb0"} Oct 30 03:11:12.562165 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:12.561310 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wd7bs/must-gather-r6s77" podStartSLOduration=2.751441304 podStartE2EDuration="3.561291396s" totalImagesPullingTime="809.850092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:12.560984554 +0000 UTC m=+6100.952792188" watchObservedRunningTime="2026-10-30 03:11:12.561291396 +0000 UTC m=+6100.953099029" Oct 30 03:11:15.206142 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:15.206073 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jt6kw_141993bb-335c-4e9c-b08e-da41dd9d7f0f/global-pull-secret-syncer/0.log" Oct 30 03:11:16.282094 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:16.281992 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lwf9z_9c859dda-f9c4-4c70-b6d2-e72b990dc465/konnectivity-agent/0.log" Oct 30 03:11:16.633656 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:16.624749 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-27.ec2.internal_0fc72893a455f4d88649494269f87258/haproxy/0.log" Oct 30 03:11:16.805141 ip-10-0-143-27 kubenswrapper[6275]: E1030 03:11:16.801303 6275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdlwv" containerName="extractor" Oct 30 03:11:17.562069 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:17.561611 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:11:18.696029 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:18.695999 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:11:18.942660 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:18.941767 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:11:19.656707 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:19.644251 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lppxm/must-gather-flqwj" event={"ID":"fb0560fc-3b1a-4834-aa3c-e69b01f245e0","Type":"ContainerStarted","Data":"8d06da3499e378b311dec8c8e351c608103972cd81c5fa868d20afc3da79895d"} Oct 30 03:11:20.376143 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:20.375221 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:11:20.668573 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:20.667411 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lppxm/must-gather-flqwj" event={"ID":"fb0560fc-3b1a-4834-aa3c-e69b01f245e0","Type":"ContainerStarted","Data":"2b6e019e6cbf706b66bdc970e8f1741a59d1647c6206858e29930a76736db8a6"} Oct 30 03:11:21.480161 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:21.478559 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:11:21.703049 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:21.700545 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:11:23.096953 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:23.095209 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:11:23.171165 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:23.171113 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:11:23.523311 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:23.523272 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7795bf7868-gmhpg_09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3/authorino/0.log" Oct 30 03:11:24.494642 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:24.493141 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:11:24.734636 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:24.734601 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:11:24.758193 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:24.752407 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:11:24.974852 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:24.974816 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:11:26.268883 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.264015 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lppxm/must-gather-flqwj" podStartSLOduration=8.76271508 podStartE2EDuration="17.263995931s" totalImagesPullingTime="8.501280851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:20.719181118 +0000 UTC m=+6109.110988751" watchObservedRunningTime="2026-10-30 03:11:26.263995931 +0000 UTC m=+6114.655803564" Oct 30 03:11:26.268883 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.265654 6275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq"] Oct 30 03:11:26.271606 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.271580 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.288972 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.288935 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq"] Oct 30 03:11:26.381961 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.381930 6275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wd7bs\"/\"default-dockercfg-mzxnn\"" Oct 30 03:11:26.404009 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.403959 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqfq\" (UniqueName: \"kubernetes.io/projected/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-kube-api-access-7rqfq\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.404264 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.404018 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-lib-modules\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.404264 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.404175 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-sys\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.404264 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.404213 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-podres\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.404264 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.404239 6275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-proc\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505163 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505106 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-sys\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505356 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505176 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-podres\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505356 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505197 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-proc\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505356 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505221 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-sys\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505356 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505305 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-proc\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505356 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505305 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-podres\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505563 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505387 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqfq\" (UniqueName: \"kubernetes.io/projected/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-kube-api-access-7rqfq\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505563 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505409 6275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-lib-modules\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.505563 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.505490 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-lib-modules\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.537714 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.537633 6275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqfq\" (UniqueName: \"kubernetes.io/projected/be70cf21-3ef1-4b79-b4f6-94eccad8b72b-kube-api-access-7rqfq\") pod \"perf-node-gather-daemonset-92mbq\" (UID: \"be70cf21-3ef1-4b79-b4f6-94eccad8b72b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.617376 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.617338 6275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:26.654253 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.654209 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:11:26.703485 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.703463 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/alertmanager/0.log" Oct 30 03:11:26.811268 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.811190 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/config-reloader/0.log" Oct 30 03:11:26.920991 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.920951 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/kube-rbac-proxy-web/0.log" Oct 30 03:11:26.951400 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:26.949854 6275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq"] Oct 30 03:11:27.033223 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.033185 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/kube-rbac-proxy/0.log" Oct 30 03:11:27.153710 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.145105 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/kube-rbac-proxy-metric/0.log" Oct 30 03:11:27.258679 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.258648 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/prom-label-proxy/0.log" Oct 30 03:11:27.369292 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.369244 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3733d077-f21f-48bf-97f4-a3bca9cae0d8/init-config-reloader/0.log" Oct 30 03:11:27.669145 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.666758 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-nfpnf_e07d2c61-a1ab-4ece-8205-6515c8b8d880/kube-state-metrics/0.log" Oct 30 03:11:27.775243 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.775169 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-nfpnf_e07d2c61-a1ab-4ece-8205-6515c8b8d880/kube-rbac-proxy-main/0.log" Oct 30 03:11:27.775500 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.775286 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" event={"ID":"be70cf21-3ef1-4b79-b4f6-94eccad8b72b","Type":"ContainerStarted","Data":"371c13908adcf74aa4c6afed2b1bc308049495cbf7273cc45e9be507fe7f5a06"} Oct 30 03:11:27.775500 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.775382 6275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" event={"ID":"be70cf21-3ef1-4b79-b4f6-94eccad8b72b","Type":"ContainerStarted","Data":"4805f9960d7363678b5f6947f9ed0222a85a87decc772f658638d82aa8341671"} Oct 30 03:11:27.775500 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.775438 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:27.841894 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.841805 6275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" podStartSLOduration=1.8417876149999999 podStartE2EDuration="1.841787615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:27.840618648 +0000 UTC m=+6116.232426281" watchObservedRunningTime="2026-10-30 03:11:27.841787615 +0000 UTC m=+6116.233595249" Oct 30 03:11:27.961306 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:27.961218 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-nfpnf_e07d2c61-a1ab-4ece-8205-6515c8b8d880/kube-rbac-proxy-self/0.log" Oct 30 03:11:28.049742 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.049693 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:11:28.218004 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.217917 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-5c7nv_19919dff-4a1d-470c-9d64-bedcb6bb5a2a/monitoring-plugin/0.log" Oct 30 03:11:28.306993 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.306958 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:11:28.721435 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.721404 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mp4k_76fcebb0-d866-476d-859e-f8b8626a4a79/node-exporter/0.log" Oct 30 03:11:28.807850 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.807820 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5vjwt_e7ec1529-3bf9-4ded-8a38-77e9b1b7964d/discovery/0.log" Oct 30 03:11:28.826038 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.826002 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mp4k_76fcebb0-d866-476d-859e-f8b8626a4a79/kube-rbac-proxy/0.log" Oct 30 03:11:28.837298 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.837263 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-m9zg8_807e1be9-1186-4cdc-a03b-7739c1720321/kube-auth-proxy/0.log" Oct 30 03:11:28.933931 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:28.933306 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mp4k_76fcebb0-d866-476d-859e-f8b8626a4a79/init-textfile/0.log" Oct 30 03:11:29.087339 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:29.087305 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-72kzd_173adfa0-45d6-4a44-b36d-f47bd184592a/istio-proxy/0.log" Oct 30 03:11:29.260373 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:29.260340 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lxtlt_b4fb616c-0b22-482d-86e1-dade8fda8955/istio-proxy/0.log" Oct 30 03:11:29.320800 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:29.320768 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-nzg5g_df283d5f-6e54-406a-b11a-9da1af27074b/istio-proxy/0.log" Oct 30 03:11:29.676589 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:29.676559 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75c985bd9-6thmd_e2362a6d-db63-4753-b0be-625d9d0be0c3/payload-processing/0.log" Oct 30 03:11:29.761024 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:29.760988 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67f4f4fc68-mtts8_01787de0-9afd-46c6-b655-2aa41bd5f89e/manager/0.log" Oct 30 03:11:30.988227 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:30.988171 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74bf55dd7c-nd7rz_38137e76-29e8-4de7-b7fc-1367a7b551a3/telemeter-client/0.log" Oct 30 03:11:31.096101 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:31.096062 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74bf55dd7c-nd7rz_38137e76-29e8-4de7-b7fc-1367a7b551a3/reload/0.log" Oct 30 03:11:31.205870 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:31.205838 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74bf55dd7c-nd7rz_38137e76-29e8-4de7-b7fc-1367a7b551a3/kube-rbac-proxy/0.log" Oct 30 03:11:31.569002 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:31.568943 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75b4575b56-vfzpw_f3452c78-90c4-4234-a319-9de51f7d97e0/manager/0.log" Oct 30 03:11:31.841760 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:31.841689 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6sv2d_ac253b0d-c298-4218-93c7-8a8787bfe66c/proxy/0.log" Oct 30 03:11:33.809987 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:33.809949 6275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-92mbq" Oct 30 03:11:34.026150 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:34.026097 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7795bf7868-gmhpg_09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3/authorino/0.log" Oct 30 03:11:35.420820 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:35.420781 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5vjwt_e7ec1529-3bf9-4ded-8a38-77e9b1b7964d/discovery/0.log" Oct 30 03:11:35.454514 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:35.454486 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-m9zg8_807e1be9-1186-4cdc-a03b-7739c1720321/kube-auth-proxy/0.log" Oct 30 03:11:35.706322 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:35.706224 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-72kzd_173adfa0-45d6-4a44-b36d-f47bd184592a/istio-proxy/0.log" Oct 30 03:11:35.902264 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:35.898513 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lxtlt_b4fb616c-0b22-482d-86e1-dade8fda8955/istio-proxy/0.log" Oct 30 03:11:35.967799 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:35.964185 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-nzg5g_df283d5f-6e54-406a-b11a-9da1af27074b/istio-proxy/0.log" Oct 30 03:11:36.202397 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:36.202368 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77d7f46558-w6nbx_070fcbc8-7d63-4a37-b073-f0b8ed68f2db/console/0.log" Oct 30 03:11:36.327402 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:36.327372 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75c985bd9-6thmd_e2362a6d-db63-4753-b0be-625d9d0be0c3/payload-processing/0.log" Oct 30 03:11:36.365677 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:36.365641 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nc6vm_ec32c8f1-eb5a-4a91-af80-f41d281d672e/download-server/0.log" Oct 30 03:11:40.632275 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:40.632235 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9gx4n_9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/dns/0.log" Oct 30 03:11:40.738798 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:40.738765 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9gx4n_9f0bb2dd-c6e3-430c-b9a2-a00b0f4a4603/kube-rbac-proxy/0.log" Oct 30 03:11:41.093928 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:41.093896 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5w55t_58b9ca87-bcc5-4b7a-8517-db3484b7c5c8/dns-node-resolver/0.log" Oct 30 03:11:41.346464 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:41.346395 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7795bf7868-gmhpg_09f2fd17-38f2-4f7d-9cec-c302c7bfdcb3/authorino/0.log" Oct 30 03:11:42.784279 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:42.784161 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5vjwt_e7ec1529-3bf9-4ded-8a38-77e9b1b7964d/discovery/0.log" Oct 30 03:11:42.814165 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:42.814134 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-m9zg8_807e1be9-1186-4cdc-a03b-7739c1720321/kube-auth-proxy/0.log" Oct 30 03:11:43.064149 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:43.064052 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-72kzd_173adfa0-45d6-4a44-b36d-f47bd184592a/istio-proxy/0.log" Oct 30 03:11:43.144404 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:43.144372 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmg28_9712dd02-a8c8-402f-bfcd-8e149dc4835b/node-ca/0.log" Oct 30 03:11:43.253261 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:43.253217 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lxtlt_b4fb616c-0b22-482d-86e1-dade8fda8955/istio-proxy/0.log" Oct 30 03:11:43.325928 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:43.324211 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-nzg5g_df283d5f-6e54-406a-b11a-9da1af27074b/istio-proxy/0.log" Oct 30 03:11:43.672002 ip-10-0-143-27 kubenswrapper[6275]: I1030 03:11:43.671924 6275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75c985bd9-6thmd_e2362a6d-db63-4753-b0be-625d9d0be0c3/payload-processing/0.log"